Compare commits

..

168 Commits

Author SHA1 Message Date
Joseph Doherty 0e5402915b fix(centralui): pending-updates rail refreshes as soon as a deployment finishes — cache drops its snapshot on Success/Failed and signals the indicator, instead of serving the pre-deploy count for up to TTL + one poll 2026-10-22 17:11:18 -04:00
Joseph Doherty 3e1435dfac docs: QA central pair relays standby inbound API calls (StandbyForwardPeerEndpoints) 2026-10-22 16:59:07 -04:00
Joseph Doherty 4bc124d73e docs: QA central pair moved to shared SQL key + secret stores; record the Windows import-rename defect 2026-10-22 15:56:25 -04:00
Joseph Doherty 6ed6800b84 fix(host): clear pooled SQLite handles before renaming the imported inbound-key file — Windows refused the rename, so the import re-ran every boot (QA vq03/vq04 2026-10-22) 2026-10-22 15:56:00 -04:00
Joseph Doherty 5ad6f83176 docs(deploy): record QA wonder-app-vq03/vq04 upgrade b3021e7c -> 3808b49f (migration 58, pair-wise window 2m35s) 2026-10-22 15:38:27 -04:00
Joseph Doherty 51b111a35f docs(deploy): record DEV wonder-app-vd03 upgrade b3021e7c -> 3808b49f (migration 58, gate 31/31) 2026-10-22 15:21:39 -04:00
Joseph Doherty 3808b49fc7 docs: record area-subtree deploy, Topology preload and the #38 rename trap in CLAUDE.md; fix stale deploy-site client-timeout line in the CLI README 2026-10-22 14:37:16 -04:00
Joseph Doherty e029dc1ccb fix(centralui): Deploy all runs in its own DI scope, live tick/Refresh/loaders pause during actions, dialog wording, failure/saved-state tests (Gitea #40) 2026-10-22 14:27:27 -04:00
Joseph Doherty bcd087efe8 feat(centralui): Topology 'Deploy all…' on site and area nodes with confirm dialog, Force and summary toast (Gitea #40) 2026-10-22 14:27:26 -04:00
Joseph Doherty d71f61e71b feat(centralui): Topology pre-loads and expands the first layer (sites -> top-level areas) (Gitea #39) 2026-10-22 14:27:26 -04:00
Joseph Doherty 1d38fb6c90 fix(cli): deploy area names a malformed summary as an invalid area deployment summary (Gitea #40 review) 2026-10-22 14:27:26 -04:00
Joseph Doherty 0956395d81 test(managementservice): MgmtDeployArea in-scope success path relays Force to a real DeploymentService and returns its summary (Gitea #40 review) 2026-10-22 14:27:26 -04:00
Joseph Doherty 74d5fba2b0 feat(cli): deploy area --id <areaId> [--force] bulk-deploys an area subtree with deploy site's deadline and exit-code contract (Gitea #40) 2026-10-22 14:27:26 -04:00
Joseph Doherty f72c72c52f feat(managementservice): MgmtDeployArea — Deployer-only, scoped by the area's owning site (fail-closed on an unknown area), long-running like MgmtDeploySite (Gitea #40) 2026-10-22 14:27:26 -04:00
Joseph Doherty eb1c9ea7f9 feat(deploymentmanager): DeployAreaAsync bulk-deploys an area subtree through the shared DeploySiteAsync core; PreviewSubtreeDeployAsync returns confirm counts (Gitea #40)
DeploySiteAsync's body moves into a private core taking the site plus an
optional area-id filter; site deploy behaviour is unchanged. The subtree is
one in-memory ParentAreaId walk (cycle-safe) shared by the deploy and the
preview. DeploymentService gains an ITemplateEngineRepository dependency for
the area lookups.
2026-10-22 14:27:26 -04:00
Joseph Doherty b9751473a6 feat(deploymentmanager): seam for subtree deploy — DeployAreaAsync / PreviewSubtreeDeployAsync signatures + SubtreeDeployPreview (Gitea #40) 2026-10-22 14:27:26 -04:00
Joseph Doherty a04eb3a752 test(playwright): token-endpoint tests tolerate DisableLogin
Under DisableLogin /auth/token deliberately answers 404 (lane K). The tests
now probe it: on a login-disabled rig TokenEndpoint_ReturnsJwt skips and the
bad-credentials test asserts 404 (the exchange is closed); with login enabled
they keep the 200 + JWT and 401 assertions.
2026-10-22 14:27:26 -04:00
Joseph Doherty 4ae3ed3fdd test(playwright): DeploymentActionTests load the lazy Topology tree via a shared helper
The Topology page has loaded lazily (bounded instance pages behind 'Load
instances' loaders) since 9ee402bb (2026-09-09); TopologyAreaTests was
updated then but DeploymentActionTests still relied on one Expand-all click
revealing instance rows, so all four facts timed out locating the row. Move
OpenStableTopologyAsync into TopologyPageHelper and use it from both.
2026-10-22 14:27:26 -04:00
Joseph Doherty 560adefb0c fix(centralui): Topology inline area rename commits the typed name on Enter (Gitea #38)
The rename input bound _renameBuffer with the default onchange trigger. A
browser fires change only after the Enter keydown (or on blur, which cancels
by design), so CommitRename always saw the original name and silently
no-oped: a UI rename never persisted. Present since the page was written
(pre-dates 513fa1aa); the Playwright test masked it by dispatching a
synthetic change event before Enter. Bind on oninput instead.

Tests: bUnit InlineRename_TypeThenEnter_PersistsNewName (fails on the old
binding); the Playwright rename test now types + Enter like an operator.
Also fix the Playwright OpenStableTopologyAsync race: it counted 'Load
instances' loaders immediately after clicking Expand all, before the
circuit re-render landed, and returned an unexpanded tree; it now waits
for no collapsed branch first.
2026-10-22 14:27:26 -04:00
Joseph Doherty 52a1e1b614 fix(configdb): dev auto-migrate uses a 30-min command timeout so large AuditLog index builds (ReviewRemediationIndexes on ~9M rows) and the peer's applock wait don't time out at 30 s 2026-10-22 10:42:35 -04:00
Joseph Doherty e7896494f1 test(site-runtime): SR5 cancel-queued-redeploy test observes which predecessor is queued instead of assuming Queued4
Deploys are compile-warmed off the actor thread and reach the four-slot
predecessor-stop queue in warm-completion order (cross-instance ordering is
not guaranteed by design). Since SR-P5 moved the config deserialization into
that warm task, the fifth-sent deploy often finishes warming before one of the
first four, takes a slot, and Queued4 terminated inside the 100 ms no-message
window (16/20 failures under parallel load). The test now waits for the four
in-flight predecessors to terminate, cancels the one still alive, and also
asserts the cancelled redeploy is answered Failed/superseded.
2026-10-22 10:35:37 -04:00
Joseph Doherty 7be22d2ac2 fix(auditlog): truncate over-width SiteCalls.LastError on the cached-telemetry dual-write and the reconciliation pull (review 2026-10-21 follow-up) 2026-10-22 09:58:46 -04:00
Joseph Doherty 89d8568fdf fix(site-runtime): a faulted seed trigger-expression eval adopts nothing instead of replaying as a real false (review 2026-10-21 follow-up L4) 2026-10-22 09:58:46 -04:00
Joseph Doherty d646f816a4 fix(site-runtime): refuse an enable while a deploy pipeline owns the instance; revert is_enabled only for an invalid name (review 2026-10-21 follow-up L3) 2026-10-22 09:58:46 -04:00
Joseph Doherty e60c316f30 fix(dcl): queue a create whose actor name belongs to a still-terminating child instead of refusing it (review 2026-10-21 follow-up L2) 2026-10-22 09:58:46 -04:00
Joseph Doherty 530a096111 fix(auditlog): advance the served-rowid mark under the write lock so a purge clamp is never overwritten stale-high (review 2026-10-21 follow-up L1) 2026-10-22 09:58:46 -04:00
Joseph Doherty 1f87942ce3 fix(auditlog): reconciliation abandons only row-intrinsic failures and only when a sibling row inserted (review 2026-10-21 follow-up M1) 2026-10-22 09:58:46 -04:00
Joseph Doherty 6b8a91cc83 docs(plans): correct the adversarial-catch descriptions in the remediation record 2026-10-22 09:52:01 -04:00
Joseph Doherty 2acd14f276 docs: CLAUDE.md review-remediation facts + 2026-10-21 remediation record 2026-10-22 09:51:52 -04:00
Joseph Doherty b67d736ac9 docs: reconcile stale cluster/security/notification docs; close register row 117, add rows 128-135 (review 2026-10-21 wave 2) 2026-10-22 09:51:52 -04:00
Joseph Doherty 46ab5594ec fix(site-call-audit): an unknown-status row no longer aborts the rest of a batched page (review 2026-10-21 06/PERF-2 follow-up) 2026-10-22 09:50:51 -04:00
Joseph Doherty 5f7f889a96 fix(notification-outbox): keyset Pending back-fill + distinct ids so a mid-claim Retry cannot double-deliver (review 2026-10-21 06/PERF-5 follow-up) 2026-10-22 09:50:51 -04:00
Joseph Doherty 1627513fdb feat(configdb): ReviewRemediationIndexes migration — AuditLog (Kind, Actor) + AuditLogEntries composite indexes
AuditLog indexes are raw SQL ON ps_AuditLog_Month so they stay partition-aligned for the switch purge.
2026-10-22 09:42:02 -04:00
Joseph Doherty 88273693f9 fix(site-call-audit): reconciliation pull serves only un-acked tracking rows (node-local ack ledger), batched page upsert, prune removed sites' cursors (review 2026-10-21 06/PERF-2 + lane H follow-up) 2026-10-22 09:38:48 -04:00
Joseph Doherty e69be1654f fix(kpi-history): fold raw samples one hour at a time; only a unique-key violation is the benign fold race (review 2026-10-21 06/PERF-6, 08/P4, 08/C3) 2026-10-22 09:38:48 -04:00
Joseph Doherty e7fb71c2de fix(site-call-audit): KPI snapshots as single conditional-aggregate queries (review 2026-10-21 06/PERF-3, 08/P3) 2026-10-22 09:38:48 -04:00
Joseph Doherty 811a561885 fix(notification-outbox): per-attempt timestamps, Pending/Retrying split claim + sweep budget, SMTP MaxConcurrentConnections clamp (review 2026-10-21 06/COR-1, 06/PERF-5, 06/IMPL-2) 2026-10-22 09:38:48 -04:00
Joseph Doherty ebf4b96789 fix(notification-service): evict stale OAuth2 token cache entries after credential rotation (review 2026-10-21 06/MEM-3) 2026-10-22 09:38:48 -04:00
Joseph Doherty 7e08a1dbfb fix(transport): import reads the template library once per session, batches instance lookups, and expands stale instances to derived/composing templates over one flatten session (review 2026-10-21 04/P4, 08/P1, 04/C2) 2026-10-22 09:38:17 -04:00
Joseph Doherty 39025b28fa fix(transport): import rejects malformed site gRPC endpoints and non-actor-safe names as blockers, not mid-apply crashes (review 2026-10-21 03/R-1, lane-F ActorNameRules follow-up) 2026-10-22 09:38:17 -04:00
Joseph Doherty 04ac4c0491 fix(transport): export reads instance children once, site connections once per site, composed templates in batches (review 2026-10-21 04/P5) 2026-10-22 09:38:16 -04:00
Joseph Doherty 56aa386784 fix(transport): serialise the export content once and pack those bytes (review 2026-10-21 04/P8a) 2026-10-22 09:38:16 -04:00
Joseph Doherty 93fbd2e030 fix(transport): pin a compressed copy of the decrypted bundle on the import session, release the upload buffer early, document the footprint (review 2026-10-21 04/M2) 2026-10-22 09:38:16 -04:00
Joseph Doherty 55cebf2397 fix(transport): zero a bundle session's content when it expires, not only on apply/abandon (review 2026-10-21 04/C3) 2026-10-22 09:38:16 -04:00
Joseph Doherty fad08f7ce7 fix(transport): remove the never-read Transport:SchemaVersionMajor option (review 2026-10-21 04/U1, 08/O3) 2026-10-22 09:38:16 -04:00
Joseph Doherty 3304381636 test(management): alarm-override test stubs the lean analysis projection the service now reads (review 2026-10-21 04/P6 integration) 2026-10-22 09:32:14 -04:00
Joseph Doherty 008976f3d8 docs(management): DebugStreamHub comments name bearer + Basic auth (review 2026-10-21 lane K follow-up) 2026-10-22 09:32:14 -04:00
Joseph Doherty 52c8a212a1 fix(management): reject native alarm source names that are not legal actor names (review 2026-10-21 lane F follow-up to 01/SR-C1) 2026-10-22 09:32:14 -04:00
Joseph Doherty df1f4efcfa fix(management): invalidate RoleMappingCache after role-mapping and scope-rule mutations commit (review 2026-10-21 lane K M1) 2026-10-22 09:32:14 -04:00
Joseph Doherty d495494074 fix(management): purge terminal PendingSecuredWrites rows on the list sweep after a retention window (review 2026-10-21 08/G3) 2026-10-22 09:32:14 -04:00
Joseph Doherty baab6ac78e fix(management): ListInstances pages, site-scopes and prefix-searches in SQL (review 2026-10-21 08/P2) 2026-10-22 09:32:14 -04:00
Joseph Doherty b0c863c6a1 fix(management): single-flight large /management bodies at the HTTP edge (409 IMPORT_IN_PROGRESS) and parse the command off the request stream (review 2026-10-21 05/M2) 2026-10-22 09:32:14 -04:00
Joseph Doherty 72f96bdbcb fix(management): compile an API method once per save — CompileForPublish replaces the validate pre-pass (review 2026-10-21 05/M1) 2026-10-22 09:32:14 -04:00
Joseph Doherty 36d57f11b3 fix(management): validate site gRPC addresses on create/update (review 2026-10-21 03/R-1) 2026-10-22 09:32:14 -04:00
Joseph Doherty a9f9ea1c9d test(host): transport fake answers the ReconcileSiteCall wrapper (review 2026-10-21 03/P-1b integration) 2026-10-22 09:15:51 -04:00
Joseph Doherty 6f6534ac7e fix(site-runtime): artifact-apply watchdog + queued-apply failure on stop, boot-skip site event, enable revert, batch Ask slack (review 2026-10-21 lane A L1-L4)
L1: an artifact apply holding the slot past 10 min (ctor-overridable) logs an
Error site event naming it without releasing the slot; PostStop answers
queued applies Failed. L2: a boot-batch instance whose actor cannot be created
emits an Error site event and leaves the deployed set. L3: a failed enable
reverts the committed is_enabled (generation-bumped both ends). L4: the routed
SetAttributes batch Ask is the 30 s per-write bound plus 10 s mailbox slack.
2026-10-22 09:10:26 -04:00
Joseph Doherty 099b2221d6 fix(dcl): ASCII-only connection actor names, guarded ActorOf, and gate rejection of colliding connection names (review 2026-10-21 lane A M2)
ConnectionActorNames.Sanitize (shared by the DCL manager and the site deploy
gate) keeps ASCII letters/digits and -_.*+:@&=,!~';() only. The manager fails a
connection whose sanitised name is invalid or already used (Error + site
event) instead of restarting; DeployCompileValidator rejects a config whose
connection names collide after sanitising.
2026-10-22 09:10:26 -04:00
Joseph Doherty 3c11510c52 fix(site-runtime): seed republished overrides into Script Actors without firing scripts; alarms still evaluate (review 2026-10-21 01/SR-U1, decision M1)
HandleOverridesLoaded now sends children a site-local AttributeValueSeed
(stream still gets the AttributeValueChanged). ScriptActor updates its
snapshot, Conditional truth value and Expression last result (seed-only eval,
never fires); AlarmActor evaluates it as a normal change.
2026-10-22 09:10:26 -04:00
Joseph Doherty 999653a6b9 perf(site-runtime): warm the next startup batch during the stagger delay and compile a batch's distinct units in parallel (review 2026-10-21 01/SR-P3)
DeployCompileValidator.WarmCompileCache dedupes script bodies/trigger
expressions across a batch and compiles them with Parallel.ForEach (half the
cores). CreateInstanceActorBatch starts the next batch's warm as it arms the
stagger timer; creation order and delay are unchanged. The warm's parsed
configs are reused at creation.
2026-10-22 09:10:26 -04:00
Joseph Doherty dde22a3dd1 perf(site-runtime): deserialize a deploy's flattened config once and hand the parsed object down (review 2026-10-21 01/SR-P5)
The deploy compile warm parses once off-thread; the parsed FlattenedConfiguration
rides DeployCompileWarmed to the gate (new Validate(FlattenedConfiguration)
overload), the redeploy buffer, CreateInstanceActor's DCL ensure and a new
InstanceActor constructor overload. Stored-row paths parse once inside
CreateInstanceActor; an unparseable stored row still fails init via S6.
2026-10-22 09:10:26 -04:00
Joseph Doherty 960c35060c perf(site-runtime): instance reconcile sweep reads metadata only and fetches config_json just for the instances it repairs (review 2026-10-21 01/SR-P1)
Adds SiteStorageService.GetAllDeployedConfigMetadataAsync. The sweep decides
from the projection, then fetches single rows for its starts/restarts off-
thread and re-validates them (generation guard, revision match, pipeline/
running guards), releasing the per-revision claim of any repair that does
not act. Boot load unchanged.
2026-10-22 09:10:26 -04:00
Joseph Doherty c782744b07 perf(site-runtime): batch Route.To().GetAttributes/SetAttributes into one Instance Actor message (review 2026-10-21 01/SR-P4)
One Ask per attribute became one site-local InstanceGet/SetAttributesRequest
answered in a single pass, keeping the per-attribute GetAttributeResponse /
SetStaticAttributeResponse outcomes. The single-write path is refactored to a
Task<SetStaticAttributeResponse> outcome shared by both paths; static writes
and rejections still reply synchronously.
2026-10-22 09:10:26 -04:00
Joseph Doherty 35ffd025c3 fix(site-runtime): serialise DeployArtifactsCommand applies per site so an overlapping apply cannot delete a live artifact (review 2026-10-21 01/SR-C2)
Each apply upserts its full set then deletes everything else, off the actor
thread; two overlapping commands interleaved and deleted artifacts central
never removed. The Deployment Manager now runs one apply at a time, queuing
the rest in arrival order; the slot is released by a self-message that lands
behind the apply's own DCL push.
2026-10-22 09:10:26 -04:00
Joseph Doherty e765700eac fix(site-runtime): republish loaded static overrides to children/stream and answer pending waits on stop (review 2026-10-21 01/SR-U1, 01/SR-M3)
SR-U1: children are created before the async override load lands, so their
trigger snapshots kept the deploy default forever after a restart/failover/
enable/sweep start. HandleOverridesLoaded now republishes every changed value
through PublishAndNotifyChildren(evaluateWaiters:false), and skips attributes
a static write touched before the load (newer than the stored row).
SR-M3: PostStop cancels each WaitForAttribute timeout and replies
'instance stopped' instead of leaving scripts to time out and timeouts to
dead-letter.
2026-10-22 09:10:26 -04:00
Joseph Doherty f5090fcd9e fix(site-runtime): reject invalid runtime actor names at the site gate and never let ActorOf escalate to the Deployment Manager (review 2026-10-21 01/SR-C1)
An instance unique name Akka refuses as a path element (space, '#', '/',
non-ASCII) made Context.ActorOf throw on the singleton's own thread; the
default Restart stopped every Instance Actor on the site. DeployCompileValidator
now checks the instance + script/alarm/native-source actor names with
ActorPath.IsValidPathElement (the Commons ActorNameRules mirror is stricter
than Akka for " ( ), pinned by a parity sweep), CreateInstanceActor catches
InvalidActorNameException and every caller reports it, and the Instance Actor
skips a refused child name like a compile failure.
2026-10-22 09:10:26 -04:00
Joseph Doherty b5cebf9fa6 fix(host): bind ConfigAuditOptions with ValidateOnStart on central (review 2026-10-21 08/G1 purge wiring) 2026-10-22 09:01:50 -04:00
Joseph Doherty c55128d83d docs(centralui): 2026-10-21 review hardening (review 2026-10-21 07, 04/C4) 2026-10-22 09:00:49 -04:00
Joseph Doherty 614aade610 fix(centralui): diagnose cache keyed on editor context; surface blocking-wait advisory as Monaco warning (review 2026-10-21 07/C-1, lane F) 2026-10-22 09:00:49 -04:00
Joseph Doherty 5c8f6ff7aa fix(centralui): bound script-analysis payload/concurrency; bound Test Run needs Deployer, site scope and audit row (review 2026-10-21 07/P-4, 07/C-2) 2026-10-22 09:00:49 -04:00
Joseph Doherty 6644cf917d fix(centralui): Topology live tick behind PollGate, refresh no longer unmounts tree (review 2026-10-21 07/P-2) 2026-10-22 09:00:49 -04:00
Joseph Doherty c429fb374b perf(centralui): load template stubs/analysis projections instead of the full tracked graph (review 2026-10-21 07/P-1, 08/P1) 2026-10-22 09:00:49 -04:00
Joseph Doherty 0ebfafa98b fix(centralui): safe-invoke fire-and-forget calls in AlarmSummary and TreeView (review 2026-10-21 07/M-4) 2026-10-22 09:00:49 -04:00
Joseph Doherty 420e934eb0 fix(centralui): cap retained event-log rows; hint keyword search 7-day window (review 2026-10-21 07/M-3, lane D follow-up) 2026-10-22 09:00:49 -04:00
Joseph Doherty b032d7fe48 fix(centralui): bound FlowDebugService script cache (review 2026-10-21 07/M-2) 2026-10-22 09:00:49 -04:00
Joseph Doherty bd0f87e523 fix(centralui): dispose ConfigurationAuditLog browser-time module (review 2026-10-21 07/M-1) 2026-10-22 09:00:49 -04:00
Joseph Doherty 0bd856552c fix(centralui): short negative TTL after a faulted pending-updates flight (review 2026-10-21 04/C4) 2026-10-22 09:00:49 -04:00
Joseph Doherty d36f360d35 fix(centralui): SessionExpiry skips redirect after dispose (review 2026-10-21 07/C-3) 2026-10-22 09:00:49 -04:00
Joseph Doherty 1e3765ba6e fix(host): NodeOptions-bound ports, central ESG cache TTL, hosted warm-up wiring; document AllowSingleNodeCluster as validation-only (review 2026-10-21 08/O5, 08/O4, 05/U3) 2026-10-22 09:00:49 -04:00
Joseph Doherty fa73ddb453 fix(inbound): background active-node-only method warm-up service (review 2026-10-21 05/U3) 2026-10-22 09:00:49 -04:00
Joseph Doherty 4921ebb0c8 fix(host): distinguish secret-store fault from missing PSK secret (review 2026-10-21 05/U2) 2026-10-22 09:00:49 -04:00
Joseph Doherty 12f815a7ea fix(esg): short-TTL definition cache for central lookups (review 2026-10-21 05/P2) 2026-10-22 09:00:49 -04:00
Joseph Doherty ee8be8001e fix(esg): JSON-serialise non-scalar GET/DELETE query parameters (review 2026-10-21 05/O2) 2026-10-22 09:00:49 -04:00
Joseph Doherty e93af5a21f fix(inbound): single-use nonce on relay provenance proofs (review 2026-10-21 05/O1) 2026-10-22 09:00:49 -04:00
Joseph Doherty 26baa5a170 fix(inbound): in-memory request buffering; relaying standby skips audit capture (review 2026-10-21 05/M4) 2026-10-22 09:00:49 -04:00
Joseph Doherty 6c9dfaf700 fix(inbound): prune expired ApiMethodCache entries; pin exact-name scopes (review 2026-10-21 05/M3) 2026-10-22 09:00:49 -04:00
Joseph Doherty d7e87c0688 fix(inbound): single-flight method compile so concurrent first callers share one assembly (review 2026-10-21 05/M1) 2026-10-22 09:00:49 -04:00
Joseph Doherty 37c25db2f1 fix(site-runtime): batch native-alarm deletes into the coalesced flush transaction; docs for scripts/cache/scheduler (review 2026-10-21 01/SR-P2) 2026-10-22 09:00:49 -04:00
Joseph Doherty 010d2ea438 fix(site-runtime): ConfigureAwait(false) on remaining script helper awaits (review 2026-10-21 01/SR-U2) 2026-10-22 09:00:49 -04:00
Joseph Doherty 4ff26a0614 fix(site-runtime): rate-limit shed Warning logs per script/alarm (review 2026-10-21 01/SR-P6) 2026-10-22 09:00:48 -04:00
Joseph Doherty 7541a38762 fix(site-runtime): reuse exited detached worker slots in script scheduler (review 2026-10-21 01/SR-M1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 96cab3f3d7 fix(site-runtime): compile cache eviction counter+warning, configurable bound, never cache faulted compiles (review 2026-10-21 01/SR-M2) 2026-10-22 09:00:48 -04:00
Joseph Doherty b0a08bb63b fix(auditlog): explicit permanent rejections on the ingest ack (IngestAck.rejected_event_ids, field 2) with a dedicated counter; reconciler ignores transient faults for abandonment; persist the served high-water mark; prune only on a non-empty enumeration (review 2026-10-21 06/PERF-1, 06/IMPL-1 adversarial follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty 38167bca88 fix(auditlog): prune per-site reconciliation state and pull-channel caches for sites absent from the enumeration; bound the abandoned-id set (review 2026-10-21 06/MEM-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 9e7e19c3c1 fix(auditlog): reconciliation pull serves only unacked rows (PushAbandoned/Rejected push-exit states), central pages within a tick with set-based inserts; refused push rows leave the queue after N attempts (review 2026-10-21 06/PERF-1, 06/COR-3, 06/IMPL-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 1da5d0bac4 fix(auditlog): page the site audit purge, bound incremental_vacuum per page, fail fast instead of parking writers on a full queue (review 2026-10-21 06/MEM-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 268b55e350 fix(auditlog): do not log a graceful-stop OperationCanceledException at Error in the telemetry drains (review 2026-10-21 06/COR-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 15496575bc fix(communication): release the _activeStreams slot when relay ActorOf throws (review 2026-10-21 03/M-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty dd60e236bd fix(communication): ReconcileSite threads the gRPC call's cancellation into central reconcile work (review 2026-10-21 03/P-1b) 2026-10-22 09:00:48 -04:00
Joseph Doherty 429ccb8d90 fix(communication): failback probe sentinel uses an illegal actor name so it can never match a real instance (review 2026-10-21 03/P-2 follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty 82976b7377 fix(communication): streaming endpoint lookups are generation-fenced and cannot strand their in-flight guards (review 2026-10-21 03/R-2 follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty e0ebebac11 fix(communication): site-address load guard cannot wedge on an unexpected fault; lifecycle-cancelled loads log at Debug (review 2026-10-21 03/M-1 follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty 4456a04e81 fix(communication): failback probation is time-boxed and cleared on address change (review 2026-10-21 03/P-2 follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty f4d328aecc fix(communication): alarm aggregator reconcile tick does not reopen while a backoff reconnect is pending (double subscribe; surfaced by review 2026-10-21 03/R-2 work) 2026-10-22 09:00:48 -04:00
Joseph Doherty c5add11546 fix(communication): command-plane failback probes readiness with a real RPC; backoff resets only after a real call succeeds (review 2026-10-21 03/P-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 017308a397 fix(communication): DebugStreamService drops the session on an InstanceNotFound snapshot (review 2026-10-21 03/M-3) 2026-10-22 09:00:48 -04:00
Joseph Doherty 90306f0ccb fix(communication): streaming consumers re-resolve site gRPC addresses on every reopen; spec matches (review 2026-10-21 03/R-2, 03/D-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 3ed71167b3 fix(communication): site-address refresh gets an overlap guard, per-load deadline and monotonic sequence (review 2026-10-21 03/M-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty b0a97a4b67 fix(communication): malformed site gRPC address keeps the old channel, never aborts the refresh tick or restarts CentralCommunicationActor (review 2026-10-21 03/R-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 0fd9bcbce6 fix(communication): delete dead SiteAddressCacheLoaded.SiteContacts + legacy NodeA/B parse (review 2026-10-21 03/D-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 004f3bd155 fix(configdb): delete dead repository members; completes 08/P8 for owned files (review 2026-10-21 08/P8; CentralUi members and tests went in the preceding P5 commit) 2026-10-22 08:33:09 -04:00
Joseph Doherty c6e4c7820a fix(configdb): InstanceLocator single untracked projected join (review 2026-10-21 08/P7) 2026-10-22 08:33:09 -04:00
Joseph Doherty 1d00d5f469 fix(configdb): deterministic config-audit paging (review 2026-10-21 08/P5) 2026-10-22 08:33:09 -04:00
Joseph Doherty 036f341a9d fix(configdb): opt-in config-audit retention purge + after-state size cap (review 2026-10-21 08/G1) 2026-10-22 08:33:09 -04:00
Joseph Doherty 394234b450 fix(dcl): advance the session generation when a reconnect attempt is issued, not when its result is applied (review 2026-10-21 02/C1 follow-up) 2026-10-22 08:33:00 -04:00
Joseph Doherty d8d44e0b87 perf(dcl): coalesce MxGateway alarm stream reopens across a subscribe burst (review 2026-10-21 02/P3) 2026-10-22 08:33:00 -04:00
Joseph Doherty e6406c494c fix(dcl): async OPC UA endpoint discovery bounded by OperationTimeoutMs/verify budget; adapter connect never runs on the actor thread (review 2026-10-21 02/P2) 2026-10-22 08:33:00 -04:00
Joseph Doherty cfe95a7e5a fix(dcl): log the initial connect to Site Event Logging like every other transition (review 2026-10-21 02/U4) 2026-10-22 08:33:00 -04:00
Joseph Doherty 3543a7f295 fix(dcl): feed ScadaBridge:MxGateway:ClientNamePrefix into the MxGateway client-name fallback (review 2026-10-21 02/U1, 08/O2) 2026-10-22 08:33:00 -04:00
Joseph Doherty f18d3adbfc fix(dcl): forget an OPC UA condition's last state once it returns to normal (review 2026-10-21 02/L3) 2026-10-22 08:33:00 -04:00
Joseph Doherty d51ea5136a fix(dcl): drop values for unsubscribed tags before quality bookkeeping; cancel the reconnect re-seed when its session ends (review 2026-10-21 02/L2, 02/L4) 2026-10-22 08:33:00 -04:00
Joseph Doherty 8c1bd2c3fc fix(dcl): advance the session generation on same-adapter reconnect and session-scope MxGateway subscription ids (review 2026-10-21 02/C1, 02/L1) 2026-10-22 08:33:00 -04:00
Joseph Doherty b811a2451b fix(dcl): remove unused poll-based WriteBatchAndWaitAsync from IDataConnection and both adapters (review 2026-10-21 02/U3) 2026-10-22 08:33:00 -04:00
Joseph Doherty 3169429c3f fix(site-event-log): queries on own read connection off the actor thread, bounded keyword window; cap trim counts/vacuums rarely (review 2026-10-21 02/P1, 02/P4) 2026-10-22 08:32:37 -04:00
Joseph Doherty f7c4adbaed docs(store-and-forward): replication text is LocalDb CDC; document lane paging + gauge reseed (review 2026-10-21 02/U4a) 2026-10-22 08:32:37 -04:00
Joseph Doherty d3cba15699 fix(store-and-forward): re-seed queue-depth gauge from DB on sweep, clamp at 0 (review 2026-10-21 02/C3) 2026-10-22 08:32:37 -04:00
Joseph Doherty 76f7960a06 fix(store-and-forward): page retry sweep per (category,target) lane; drop dead GetAllMessagesAsync (review 2026-10-21 02/C2, 02/U2) 2026-10-22 08:32:37 -04:00
Joseph Doherty e968566e9d fix(cli): no redirects on the management client, race-safe token refresh, verbose Basic-fallback note (review 2026-10-21 05/U1, 05/P1 L1/L2/L3/L4)
- CreateDefaultHttpClient sets AllowAutoRedirect=false so a 307/308 cannot re-POST the
  /auth/token credential form elsewhere.
- A forced re-exchange after a 401 carries the refused token; inside the gate it reuses
  a token another caller already re-exchanged instead of binding again. The BasicOnly
  switch is written under the same gate.
- SCADABRIDGE_VERBOSE=1 prints a stderr note when a fresh token is refused and the CLI
  falls back to Basic.
- debug stream's AccessTokenProvider retries the exchange once and documents the
  null-token degradation (the hub refuses, automatic reconnect retries).
- Tests: concurrent refusal reuses the peer refresh (2 exchanges, not 3); real-socket
  redirect is not followed.
2026-10-22 08:25:35 -04:00
Joseph Doherty ea172edebc fix(security): bearer token hardening follow-ups — nbf skew, TTL-0 path, cache docs, extra rejection tests (review 2026-10-21 05/U1, 05/P1 L1/L5/L6)
- JwtTokenService back-dates nbf by the 30 s validation skew so a peer central whose
  clock lags the issuer accepts a fresh token after failover.
- RoleMappingCacheSeconds = 0 routes RoleMapper to the matched-rows loader (old cost)
  instead of loading every Deployer mapping's scope rules per call.
- RoleMappingCache / SecurityOptions / Component-Security.md describe Invalidate() by
  its contract plus the TTL, not by a call site.
- Tests: HS512 with the same key, issuer/audience mismatch, lagging-peer nbf, TTL-0
  matched-only reads, refresh-vs-Invalidate generation race.
2026-10-22 08:25:35 -04:00
Joseph Doherty 1b3928a0a2 fix(security): finish the bearer-token design and cache role mappings (review 2026-10-21 05/U1, 05/P1)
- ManagementAuthenticator accepts Authorization: Bearer <jwt> alongside Basic on
  /management, the audit REST endpoints and the debug-stream hub. Tokens are
  validated locally (HS256 only, alg=none rejected, issuer/audience, lifetime with
  30 s skew); roles and site scope inside a valid token are authoritative for its
  lifetime, so a bearer call costs no LDAP bind and no role re-map.
- /auth/token returns expires_in + management_bearer:true, carries username, display
  name, roles and permitted sites, and answers 404 under DisableLogin.
- CLI exchanges credentials once per process (in-memory CliTokenStore), sends Bearer,
  re-exchanges once on 401 then falls back to Basic; older/login-disabled servers are
  Basic-only; bad credentials are not retried with Basic. Debug stream uses the token
  via AccessTokenProvider. SCADABRIDGE_AUTH_MODE=basic opts out.
- StartupValidator fails a Central boot on a JwtSigningKey under 32 UTF-8 bytes;
  fixed the false 'fail-fast at construction' comment. Removed the dead
  RefreshToken/RecordActivity/ShouldRefresh/IsIdleTimedOut members and the inert
  JwtRefreshThresholdMinutes; JwtExpiryMinutes now documented as the bearer lifetime
  (validated > 0).
- RoleMapper reads through a process-wide RoleMappingCache (RoleMappingCacheSeconds,
  default 30 s): one refresh loads the table plus every Deployer mapping's scope
  rules, Invalidate() drops it (generation-guarded against in-flight refreshes).
- Component-Security.md / Component-CLI.md / CLI README describe the real model
  (cookie claims ticket, bearer for CLI) instead of a cookie-embedded JWT.
2026-10-22 08:25:35 -04:00
Joseph Doherty f0de11c446 fix(centralui-auth): memoize non-authoritative circuit auth-state reads for 10 s (review 2026-10-21 07/P-3)
GetAuthenticationStateAsync (display reads: current username, nav role checks) used
to force a full revalidation + DB role re-map on every call. It now reuses the
circuit's last revalidation for NonAuthoritativeRefreshInterval (10 s);
GetAuthoritativeStateAsync (mutation gates, UI-13) still re-maps every time and
fails closed. The revocation regression test now asserts through the authoritative
read, which is what every gate uses.
2026-10-22 08:25:35 -04:00
Joseph Doherty 77b3a2c23f fix(commons): ActorNameRules accepts ( ) and " like Akka's IsValidPathElement (review 2026-10-21 01/SR-C1 parity) 2026-10-22 08:24:48 -04:00
Joseph Doherty 7aacc49043 fix(deployment): detach each instance's settled rows from the bulk deploy's shared unit of work after Finalize (review 2026-10-21 04/M3) 2026-10-22 08:22:42 -04:00
Joseph Doherty 32e63eaa96 fix(deployment): bulk-deploy up-to-date pre-filter uses batch hash/status reads and the operation's shared flatten session; completion log reports pre-filter cost (review 2026-10-21 04/P1) 2026-10-22 08:22:42 -04:00
Joseph Doherty df67fabe68 fix(deployment): pending-updates tick uses a slim instance projection, hash-only flatten, one untracked mismatch snapshot read and a per-tick secret lookup memo (review 2026-10-21 04/P2) 2026-10-22 08:22:42 -04:00
Joseph Doherty 971a9886b8 fix(deployment): drop dead ITemplateGraphWatermark ctor param and no-op ClearMemos from StaleInstanceProbe (review 2026-10-21 04/U2) 2026-10-22 08:22:42 -04:00
Joseph Doherty 7975699ac4 fix(deployment): flatten reads compositions off the loaded template instead of a second per-template query (review 2026-10-21 04/U5) 2026-10-22 08:22:42 -04:00
Joseph Doherty 669822772e fix(deployment): read every external system's methods in one query for the artifact snapshot (review 2026-10-21 04/P7) 2026-10-22 08:22:41 -04:00
Joseph Doherty 58b1e021b2 fix(deployment): drop unused INotificationRepository from ArtifactDeploymentService (review 2026-10-21 04/U4) 2026-10-22 08:22:41 -04:00
Joseph Doherty 7c83a4aa94 fix(reconcile): batch startup reconcile into fixed round trips, bound central work by QueryTimeout, retry failed site passes with back-off (review 2026-10-21 03/P-1) 2026-10-22 08:22:41 -04:00
Joseph Doherty d6a9a79c27 fix(deployment): purge SystemArtifactDeploymentRecords in the terminal-record retention sweep (review 2026-10-21 08/G2) 2026-10-22 08:22:41 -04:00
Joseph Doherty da5a049077 fix(deployment): pending-deployment supersede/purge/stage deletes never read ConfigurationJson; drop stale deferred-TODO comment (review 2026-10-21 08/P6, 08/C2) 2026-10-22 08:22:41 -04:00
Joseph Doherty 0d4f7d33b9 perf(templateengine): memoise CollisionDetector's composed-module walk per check (review 2026-10-21 04/P8b) 2026-10-22 06:33:44 -04:00
Joseph Doherty 52aab37299 fix(templateengine): audit projected template/script summaries instead of the tracked graph and script bodies (review 2026-10-21 08/G1) 2026-10-22 06:33:44 -04:00
Joseph Doherty c0792cff3e fix(templateengine): delete dead SiteService data-connection create/update that audited raw credentials (review 2026-10-21 08/C1) 2026-10-22 06:33:44 -04:00
Joseph Doherty 4102922640 docs(templateengine): drop the removed staleness memo; describe the full shared-script compile gate (review 2026-10-21 04/U2, 04/U3) 2026-10-22 06:33:44 -04:00
Joseph Doherty c7774b7ec2 perf(templateengine): alarm-override, folder-delete and template-delete checks read lean template projections (review 2026-10-21 04/P6, 08/P1)
InstanceService.SetAlarmOverrideAsync and TemplateDeletionService move to the
no-tracking, body-free GetAllTemplatesForAnalysisAsync; TemplateFolderService
counts folder contents from GetTemplateStubsAsync.
2026-10-22 06:33:44 -04:00
Joseph Doherty 7d11a1db8d fix(templateengine): reject instance/script/alarm/slot names that are not valid actor names at save (review 2026-10-21 01/SR-C1) 2026-10-22 06:33:44 -04:00
Joseph Doherty 5919a71b8b fix(scriptanalysis): warn on scripts blocking on a Task (.Result/.Wait()/GetAwaiter().GetResult()) (review 2026-10-21 01/SR-U2)
Advisory, not a trust-policy violation: every policy entry blocks deploy, and
an error here would fail the next redeploy of already-deployed scripts.
ValidationService surfaces it as a non-blocking ScriptCompilation warning.
2026-10-22 06:33:44 -04:00
Joseph Doherty 2ca60f86ff test(templateengine): drop racy verdict-cache Count assertion from the faulted-compile test (review 2026-10-21 04/C1) 2026-10-22 06:33:44 -04:00
Joseph Doherty 8c935480fa fix(scriptanalysis): memoise ScriptTrustValidator verdicts by code hash for the default reference set (review 2026-10-21 04/P3) 2026-10-22 06:33:44 -04:00
Joseph Doherty ae1282ed9d fix(scriptanalysis): compile gate reads diagnostics only and never caches a faulted verdict (review 2026-10-21 04/M1, 04/C1)
RoslynScriptCompiler.Compile now uses Script.GetCompilation().GetDiagnostics()
with one shared, never-loading InteractiveAssemblyLoader instead of
Script.Compile(), which emitted and loaded a non-collectible assembly (and a
fresh AssemblyLoadContext) per distinct script. The catch-all that turned any
exception into a 'Compilation exception' diagnostic is gone; TryCompile and
CheckExpressionSyntax report a fault for that call only, uncached.
2026-10-22 06:33:44 -04:00
Joseph Doherty 75149e83b0 fix(deps): clear new NuGet audit advisories (Pkcs, Cryptography.Xml, SignalR.Common)
Targeted transitive pins per the house pattern, no family-wide bump, no suppression:
Cryptography.Xml 10.0.10->10.0.13; Pkcs pinned 10.0.13 in ConfigurationDatabase,
ExternalSystemGateway, NotificationService, Playwright tests; SignalR.Common 10.0.13 in CLI.
2026-10-22 06:15:25 -04:00
Joseph Doherty 2da90bf151 chore(host): remove unused Database:MachineDataDb option, validation, config and docs 2026-10-22 06:08:18 -04:00
Joseph Doherty bc1aca0ee6 feat(instances): GetInstancesPagedAsync SQL-side filter/order/page 2026-10-22 06:06:27 -04:00
Joseph Doherty 6bcca4b344 feat(templates): GetTemplateStubsAsync untracked identity/edge projection 2026-10-22 06:06:27 -04:00
Joseph Doherty 15c9de0f4a feat(commons): GrpcEndpointRules for site gRPC address validation 2026-10-22 06:05:06 -04:00
Joseph Doherty 3a53e7059b feat(commons): ActorNameRules mirroring Akka path-element validation 2026-10-22 06:05:06 -04:00
Joseph Doherty 513fa1aa16 docs(tests): fix CommentChecker findings across tests/ — 13,392 → 0 (repo-wide scan clean)
Comment/XML-doc changes only:
- add summaries to ~10k test methods, fixtures, fakes and helpers
  (+ <param> on theories, <returns> on async Task methods,
  <inheritdoc/> on interface implementations the checker can resolve)
- strip ~2.6k project-tracking references (WP*, P*-*, row-NN, Task N,
  #NN, Component-0NN) from comments, keeping the explanatory text;
  reword false positives in comments only
- real docs on fakes implementing BCL/library interfaces
  (IDisposable, ILogger, IOptionsMonitor, IAsyncLifetime)
- reword the last 5 src/ false positives (CWE-1236, failures-1,
  year-0001, M365)

Only non-comment edits: four one-line nested classes split across lines
so members can carry docs. Build 0 errors / 0 warnings.
2026-10-07 15:05:49 -04:00
Joseph Doherty 0ffd4bdbea docs(src): fix CommentChecker findings across src/ — 1,032 → 5 (all false positives)
Comment/XML-doc changes only, no code logic:
- strip project-tracking references (WP*, P*-*, row-NN, #NN, Task N) from
  comments, keeping the explanatory rationale
- add missing summaries, <param>, <returns>, <typeparam> and <inheritdoc/>
- replace misused <inheritdoc/> with real docs; where <inheritdoc/> was
  combined with useful remarks, move the rationale into // comments
- drop stale replicationActor <param> on DeploymentManagerActor ctor

Remaining 5 are false positives (CWE-1236 x2, failures-1, year-0001, M365).
Build 0 errors / 0 warnings.
2026-10-07 14:12:25 -04:00
Joseph Doherty 3a217b9484 fix(site-secrets): keep a live node's mirror-status row from losing a race with the peer's prune
Rig finding on the upgrade restart: both nodes' rows were older than the stale threshold
(the old build never refreshed them), so node-a's reporter pruned node-b's row while node-b
wrote a fresh one, and the replicated delete won. The reporter now prunes nothing during its
first refresh interval after start-up, and each mirror pass re-writes its own row if it no
longer holds what the mirror last wrote.
2026-09-29 11:06:26 -04:00
Joseph Doherty 55e1741075 fix(site-secrets): prune secret_mirror_status rows of decommissioned or renamed nodes
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval
(10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and
deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the
refresh). The delete is conditional on the checked_at it read, so a node that came back
keeps its fresh row; a returning node re-writes its row on its first pass.
2026-09-29 10:56:53 -04:00
Joseph Doherty 2650f79472 test(rig): site secrets gate covers the follow-ups — standby KEK mismatch on the health report, removal audit, CLI exit codes — PASS 9/9
Also: DeploymentDeadlineParityTests' stub replies with a real zero-failure summary (the artifact
aliases now derive their exit code from failureCount), and register row 127 reflects the merge
and the two new replicated site tables.
2026-09-28 21:46:49 -04:00
Joseph Doherty 4e6409184c Merge branch 'fix/standby-secret-mirror-status' (M1: standby mirror failures on the health report)
# Conflicts:
#	CLAUDE.md
#	docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md
2026-09-28 21:37:03 -04:00
Joseph Doherty f5756affef docs(site-secrets): document secret_mirror_status and standby mirror failures on the health report (M1) 2026-09-28 21:33:10 -04:00
Joseph Doherty 0289d51137 fix(site-secrets): report every site node's secret mirror failures on the health report (M1)
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.

- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
  failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
  mirror writes only its own row, and only when its failure set changes plus
  once per start-up (every write replicates via CDC). Best-effort: a failed
  write is a Warning and is retried next pass. Node key = INodeIdentityProvider
  NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
  into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
  SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
  null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
  node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
  round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
  bUnit, replicated-table set (twelve) + FK guard wording.
2026-09-28 21:33:10 -04:00
Joseph Doherty 253939cd5d docs(site-secrets): Secrets 0.6.3 is published (not consumed by ScadaBridge) 2026-09-28 21:31:05 -04:00
Joseph Doherty 29049c4ef6 fix(site-secrets): contain secret-store outages per site; audit removal-only deploys
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
2026-09-28 21:23:52 -04:00
Joseph Doherty ef03a0dca7 fix(cli): artifact-deploy commands exit 1 when any site failed
site deploy-artifacts and deploy artifacts rendered the per-site summary and
always exited 0, so a script could not tell a failed site (unreachable,
missing or unreadable secret, partial apply) from a clean deploy. Both now
route through the failureCount rule deploy site already used.
2026-09-28 21:23:52 -04:00
1464 changed files with 49077 additions and 6926 deletions
+14 -12
View File
File diff suppressed because one or more lines are too long
+18 -1
View File
@@ -45,6 +45,7 @@
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.TestHost" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.SignalR.Client" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.SignalR.Common" Version="10.0.13" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Scripting" Version="5.0.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="5.0.0" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="6.1.1" />
@@ -164,9 +165,25 @@
(NU1605 downgrade errors) that belongs in its own reviewed commit.
-->
<ItemGroup>
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.13" />
<PackageVersion Include="System.Security.Cryptography.Pkcs" Version="10.0.13" />
</ItemGroup>
<!--
2026-10-22 NuGet audit refresh (fresh restore went red under TreatWarningsAsErrors):
- System.Security.Cryptography.Xml 10.0.10 now carries GHSA-7hqp-7c5m-4vq6,
GHSA-c399-9853-6jmj, GHSA-r955-8q5j-3h83 (high): pin raised 10.0.10 -> 10.0.13.
- System.Security.Cryptography.Pkcs (GHSA-2fmv-rv2c-59x7 moderate, GHSA-846f-f764-76mx
high) reaches the graph as 9.0.4 / 10.0.0 / 10.0.10 via SqlClient 6.1.1, MimeKit and
System.Security.Cryptography.Xml: pinned to 10.0.13 with explicit PackageReferences in
ConfigurationDatabase, ExternalSystemGateway, NotificationService and the Playwright tests,
the projects where those chains enter.
- Microsoft.AspNetCore.SignalR.Common 10.0.7 (GHSA-334f-vvp4-7g6p, high) comes in
transitively through SignalR.Client in the CLI: pinned to 10.0.13 in the CLI only.
Same house pattern as above: targeted transitive pins, no family-wide servicing bump, no
NuGetAuditSuppress/NoWarn.
-->
<!--
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
+2 -2
View File
@@ -18,7 +18,7 @@ This repository is the full **implementation** project for ScadaBridge — the C
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
| Hosting | Windows Server, Windows Service |
| Cluster | Akka.NET Cluster (active/standby, keep-oldest SBR) |
| Cluster | Akka.NET Cluster (active/standby, `auto-down` downing) |
| Logging | Serilog (structured) |
### Scale
@@ -88,7 +88,7 @@ Both stacks share the infrastructure services in [`infra/`](infra/) (MS SQL, LDA
| 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. |
| 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
| 12 | Site Event Logging | [docs/requirements/Component-SiteEventLogging.md](docs/requirements/Component-SiteEventLogging.md) | SQLite storage, 30-day retention + 1GB cap, daily purge, paginated remote queries with keyword search. |
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, keep-oldest SBR with down-if-alone, min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/SBR/CoordinatedShutdown wiring lives in the Host. |
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, `auto-down` downing strategy (2026-07-21 availability-over-partition-safety decision; `keep-oldest` stays a supported option), min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/downing/CoordinatedShutdown wiring lives in the Host. |
| 14 | Inbound API | [docs/requirements/Component-InboundAPI.md](docs/requirements/Component-InboundAPI.md) | POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object/List), script-based implementations, failures-only logging. |
| 15 | Host | [docs/requirements/Component-Host.md](docs/requirements/Component-Host.md) | Single deployable binary, role-based component registration, per-component config binding (Options pattern), readiness gating, dead letter monitoring, Akka.NET bootstrap, ASP.NET Core hosting for central. |
| 16 | Commons | [docs/requirements/Component-Commons.md](docs/requirements/Component-Commons.md) | Namespace/folder convention (Types/Interfaces/Entities/Messages), shared data types, POCOs, repository interfaces, message contracts with additive-only versioning, UTC timestamp convention, the unified read-only alarm condition model (`AlarmConditionState`/`AlarmKind`), and native alarm source entities + the `IAlarmSubscribableConnection` capability seam. |
@@ -27,7 +27,7 @@
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
"_comment": "ConfigurationDb (which carries the dev SQL password) is supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
@@ -27,7 +27,7 @@
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
"_comment": "ConfigurationDb (which carries the dev SQL password) is supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
-2
View File
@@ -47,7 +47,6 @@ services:
# that SAME shared mssql container. Real deployments use ${secret:} tokens +
# a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# LDAP bind credential for the redundant local GLAuth pair (scadaproj/infra/glauth-redundant,
# host.docker.internal:3893/:3894 — see central-node-a/appsettings.Central.json). Same
# value docker/'s central nodes use for the SAME local pair; the shared 10.100.0.35
@@ -110,7 +109,6 @@ services:
# that SAME shared mssql container. Real deployments use ${secret:} tokens +
# a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# LDAP bind credential for the redundant local GLAuth pair (scadaproj/infra/glauth-redundant,
# host.docker.internal:3893/:3894 — see central-node-b/appsettings.Central.json). Same
# value docker/'s central nodes use for the SAME local pair; the shared 10.100.0.35
@@ -27,7 +27,7 @@
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) \u2014 this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
"_comment": "ConfigurationDb (which carries the dev SQL password) is supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) \u2014 this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
@@ -27,7 +27,7 @@
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) \u2014 this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
"_comment": "ConfigurationDb (which carries the dev SQL password) is supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) \u2014 this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
-2
View File
@@ -107,7 +107,6 @@ services:
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
@@ -174,7 +173,6 @@ services:
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) — consolidation, not new exposure. Real
# deployments use ${secret:} tokens (src/.../appsettings.Central.json) + a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
+111 -30
View File
@@ -22,19 +22,24 @@ set -euo pipefail
# Ciphertext is compared by SHA-256 only; secret VALUES are random throwaways held in shell
# variables and handed to the browser through the environment — never printed, never on argv.
#
# Checks (each prints PASS/FAIL; the run ends "PASS n/8" and exits non-zero on any FAIL):
# Checks (each prints PASS/FAIL; the run ends "PASS n/9" and exits non-zero on any FAIL):
# 1 secret + secret-referencing MxGateway connection on site-a -> site-a Out of date, b/c Current
# 2 deploy site-a -> success, secretsDeployed 1, site-a Current
# 3 both site-a nodes hold the secret at the central revision + ciphertext (standby <= 30 s)
# 4 rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision
# 5 drop the secret: reference -> deploy -> both nodes no longer hold it; an out-of-band secret
# seeded on site-a-a through the library's own write path (the ZB.MOM.WW.Secrets `secret` CLI
# run against that node's store) survives
# 6 reference a non-existent secret -> site-a deploy fails closed with the named message,
# site-a stays Out of date, nothing changed on either site-a node
# 7 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the KekId
# 8 one DeploySiteSecrets audit row per secret-bearing deploy; no plaintext / base64 ciphertext
# in any of the 8 nodes' logs
# 5 standby master-key mismatch reaches central (review M1): site-a-b is recreated with a WRONG
# KEK -> central's site-a health report shows node-b failing to decrypt the secret while
# node-a is clean; site-a-b restored with the right KEK -> node-b's failures clear
# 6 drop the secret: reference -> deploy -> both nodes no longer hold it and the reply names it
# in secretsRemoved; an out-of-band secret seeded on site-a-a through the library's own write
# path (the ZB.MOM.WW.Secrets `secret` CLI run against that node's store) survives
# 7 reference a non-existent secret -> site-a deploy fails closed with the named message, the
# CLI exits 1, site-a stays Out of date, nothing changed on either site-a node
# 8 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the
# KekId, the CLI exits 1
# 9 one DeploySiteSecrets audit row per deploy that carried, lacked or retired a secret (the
# removal-only deploy of check 6 included, with its removed list); no plaintext / base64
# ciphertext in any of the 8 nodes' logs
#
# Teardown (trap EXIT, also run up-front so a rerun starts clean): gate connections deleted,
# site-a + site-b redeployed so the removal reaches them, the central gate secret deleted, the
@@ -56,6 +61,10 @@ A=(--url "$URL" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" -
SECRETS_CLI_PROJECT="${SECRETS_CLI_PROJECT:-$HOME/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli}"
SECRETS_CLI_DIR="${SECRETS_CLI_DIR:-${TMPDIR:-/tmp}/sb-gate-secrets-cli}"
STANDBY_WINDOW_S="${STANDBY_WINDOW_S:-30}"
# Check 5: mirror pass (15 s) + status reporter poll (30 s) + health report (30 s), plus a restart.
HEALTH_WINDOW_S="${HEALTH_WINDOW_S:-240}"
# The site-b/site-c dev KEK (docker-compose.yml) — deliberately NOT site-a's, used by check 5.
WRONG_KEK="esKrmqV2RBmMq/+6T/5zK2BcalRy5gzDXhoG9Qydg6Q="
# DEV-ONLY rig KEK shared by central + site-a (docker/docker-compose.yml, x-secrets-shared-kek-env).
SITE_A_KEK="zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
@@ -333,6 +342,39 @@ v=None if r is None else r.get(sys.argv[2])
print("" if v is None else (json.dumps(v) if isinstance(v,bool) else v))' "$1" "$2"
}
badge_of() { jget "$1" "$2"; }
# mirror_failures <siteIdentifier> -> JSON {nodeName: [secretName...]} from central's latest health
# report for the site (SiteHealthReport.SecretMirrorStatuses), or "null" when not reported
mirror_failures() {
"$CLI" health site "${A[@]}" --identifier "$1" 2>/dev/null | python3 -c '
import json,sys
try:
r=(json.load(sys.stdin).get("latestReport") or {}).get("secretMirrorStatuses")
except Exception:
r=None
print("null" if r is None else json.dumps({n["nodeName"]: sorted(f["secretName"] for f in n["failures"]) for n in r}, sort_keys=True))'
}
# The rig's compose project dir (where the bind-mounted appsettings live), from the running container.
RIG_DIR=$(docker inspect "$STANDBY_A" --format '{{ index .Config.Labels "com.docker.compose.project.working_dir" }}' 2>/dev/null)
# recreate_standby_a [wrong] -> recreates site-a-b from the rig's own compose file, optionally with
# an override that swaps in a mismatched KEK; waits until it is a cluster member again.
recreate_standby_a() {
local files=(-f "$RIG_DIR/docker-compose.yml")
if [ "${1:-}" = "wrong" ]; then
printf 'services:\n site-a-b:\n environment:\n ZB_SECRETS_MASTER_KEY: "%s"\n' "$WRONG_KEK" >"$WORK/wrong-kek.yml"
files+=(-f "$WORK/wrong-kek.yml"); touch "$WORK/standby-kek-swapped"
fi
docker compose "${files[@]}" --project-directory "$RIG_DIR" up -d --no-deps --force-recreate site-a-b >/dev/null 2>&1 || return 1
[ "${1:-}" = "wrong" ] || rm -f "$WORK/standby-kek-swapped"
local i
for i in $(seq 1 60); do
docker run --rm --network "container:$ACTIVE_A" python:3.13-alpine python3 -c '
import urllib.request as u,sys
try: sys.exit(0 if u.urlopen("http://scadabridge-site-a-b:8084/health/ready",timeout=3).status==200 else 1)
except Exception: sys.exit(1)' >/dev/null 2>&1 && return 0
sleep 3
done
return 1
}
ensure_secrets_cli() {
[ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ] && return 0
@@ -365,6 +407,9 @@ teardown() {
echo ""
echo "=== teardown ==="
local id
if [ -f "$WORK/standby-kek-swapped" ]; then
recreate_standby_a && echo " restored site-a-b with site-a's KEK" || echo " WARN could not restore site-a-b's KEK"
fi
for c in "$CONN_A" "$CONN_B"; do
id=$(conn_id "$c" 2>/dev/null)
if [ -n "$id" ]; then
@@ -493,7 +538,37 @@ B=$(gp badges "$URL")
end_check 4
# ── 5 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 5 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
begin_check 5 "standby master-key mismatch reaches central's health report (site-a-b on a wrong KEK)"
# wait_health <python predicate over F = {node: [failing secret names]}> -> sets HEALTH_S / LAST_F
wait_health() {
local start now; start=$(python3 -c 'import time; print(time.time())'); HEALTH_S=""
while :; do
LAST_F=$(mirror_failures site-a)
if python3 -c "import json,sys; F=json.loads(sys.argv[1]); sys.exit(0 if F is not None and ($1) else 1)" "$LAST_F"; then
now=$(python3 -c 'import time; print(time.time())'); HEALTH_S=$(python3 -c "print(round($now-$start))"); return 0
fi
now=$(python3 -c 'import time; print(time.time())')
python3 -c "import sys; sys.exit(0 if $now-$start < $HEALTH_WINDOW_S else 1)" || return 1
sleep 5
done
}
if wait_health "F.get('node-a')==[] and F.get('node-b')==[]"; then ok "baseline: central reports both site-a nodes, no failures ($LAST_F)"
else bad "baseline: site-a health report is not two clean node rows: $LAST_F"; fi
if [ -z "$RIG_DIR" ]; then bad "cannot find the rig's compose directory from $STANDBY_A's labels"
elif recreate_standby_a wrong; then
ok "site-a-b recreated with a mismatched KEK and back in the cluster (standby; site-a-a stays active)"
if wait_health "'$SECRET' in F.get('node-b',[]) and F.get('node-a')==[]"; then MISMATCH_S="$HEALTH_S"
ok "central's site-a report shows node-b failing $SECRET, node-a clean (${HEALTH_S}s): $LAST_F"
else bad "node-b's failure never reached central within ${HEALTH_WINDOW_S}s: $LAST_F"; fi
if recreate_standby_a; then ok "site-a-b restored with site-a's KEK"; else bad "could not restore site-a-b"; fi
if wait_health "F.get('node-a')==[] and F.get('node-b')==[]"; then CLEARED_S="$HEALTH_S"
ok "node-b's failure cleared on central's report (${HEALTH_S}s)"
else bad "node-b's failure did not clear within ${HEALTH_WINDOW_S}s: $LAST_F"; fi
else bad "could not recreate site-a-b with a mismatched KEK"; fi
end_check 5
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 6 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
SB_GATE_VALUE="$VALUE_OOB" site_a_secret_set "$OOB" "site-secrets gate out-of-band" >/dev/null 2>&1 || true
J=$(gp site-secrets "$ACTIVE_A")
if [ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ]; then
@@ -503,6 +578,7 @@ upsert_conn "$CONN_A" "$SITE_A_ID" "gate-literal-not-a-secret-ref" && ok "$CONN_
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "deploy succeeded (secretsDeployed='$(site_result site-a secretsDeployed)')" \
|| bad "deploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
case "$(site_result site-a secretsRemoved)" in *"$SECRET"*) ok "reply names $SECRET in secretsRemoved";; *) bad "secretsRemoved='$(site_result site-a secretsRemoved)', expected $SECRET";; esac
GONE="(d.get('$SECRET') is None or d['$SECRET']['deleted'])"
if wait_node "$ACTIVE_A" "$GONE" "$T_DEPLOYED"; then ok "site-a-a no longer holds $SECRET (${WAIT_S}s)"; else bad "site-a-a still holds $SECRET: $LAST_NODE_JSON"; fi
if wait_node "$STANDBY_A" "$GONE" "$T_DEPLOYED"; then STANDBY_5="$WAIT_S"; ok "site-a-b no longer holds $SECRET (${WAIT_S}s after the deploy returned)"
@@ -512,10 +588,10 @@ J=$(gp site-secrets "$ACTIVE_A")
&& ok "out-of-band $OOB still live on site-a-a" || bad "out-of-band $OOB lost: $J"
L=$(gp site-localdb "$ACTIVE_A")
[ "$(jget "$L" "deployed.$SECRET.removed")" = "true" ] && ok "deployed_secrets.$SECRET is_removed=1 (kept for the standby)" || bad "deployed_secrets: $L"
end_check 5
end_check 6
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 6 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 7 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
SNAP_BEFORE="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$MISSING" && ok "$CONN_A now references secret:$MISSING" || bad "update $CONN_A"
deploy "$SITE_A_ID"
@@ -523,6 +599,7 @@ ERR=$(site_result site-a errorMessage)
echo " deploy rc=$DEPLOY_RC error: $ERR"
EXPECT="Secret '$MISSING' referenced by data connection(s) $CONN_A is not in the central secret store"
[ "$(site_result site-a success)" = "false" ] && ok "site-a deploy reported failure" || bad "site-a deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
[ "$DEPLOY_RC" -eq 1 ] && ok "CLI exited 1 (failureCount 1)" || bad "CLI exited $DEPLOY_RC, expected 1"
case "$ERR" in *"$EXPECT"*) ok "named fail-closed message";; *) bad "message does not contain: $EXPECT";; esac
B=$(gp badges "$URL"); echo " badges: $B"
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a stays Out of date" || bad "site-a is '$(badge_of "$B" site-a)'"
@@ -530,10 +607,10 @@ sleep 3
SNAP_AFTER="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
[ "$SNAP_BEFORE" = "$SNAP_AFTER" ] && ok "nothing changed: stamped hash, both nodes' secret rows, deployed_secrets, ledgers and the $CONN_A definition identical" \
|| { bad "site state changed"; echo " before: $SNAP_BEFORE"; echo " after: $SNAP_AFTER"; }
end_check 6
end_check 7
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 7 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 8 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
C=$(gp central-secret "$SECRET")
KEK=$(jget "$C" kek)
upsert_conn "$CONN_B" "$SITE_B_ID" "secret:$SECRET" && ok "created $CONN_B on site-b (ApiKey=secret:$SECRET)" || bad "create $CONN_B"
@@ -541,13 +618,14 @@ deploy "$SITE_B_ID"
ERR=$(site_result site-b errorMessage)
echo " deploy rc=$DEPLOY_RC error: $ERR"
[ "$(site_result site-b success)" = "false" ] && ok "site-b deploy reported failure" || bad "site-b deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
[ "$DEPLOY_RC" -eq 1 ] && ok "CLI exited 1 (failureCount 1)" || bad "CLI exited $DEPLOY_RC, expected 1"
case "$ERR" in *"$SECRET"*"$KEK"*) ok "message names $SECRET and the row's KekId $KEK";; *) bad "message does not name $SECRET + KekId $KEK";; esac
B=$(gp badges "$URL"); echo " badges: $B"
[ "$(badge_of "$B" site-b)" = "out of date" ] && ok "site-b not stamped (Out of date)" || bad "site-b is '$(badge_of "$B" site-b)'"
end_check 7
end_check 8
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 8 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
# ── 9 ────────────────────────────────────────────────────────────────────────────────────────
begin_check 9 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
AUD=$("$CLI" audit-config query "${A[@]}" --action DeploySiteSecrets --entity-type SiteSecrets --from "$GATE_START" --page-size 50 2>/dev/null)
ROWS=$(printf '%s' "$AUD" | python3 -c '
import json,sys
@@ -557,18 +635,20 @@ for e in sorted(d.get("entries",[]), key=lambda e: e["timestamp"]):
s=json.loads(e.get("afterStateJson") or "{}")
names=",".join("%s@r%s" % (x.get("name"), x.get("revision")) for x in (s.get("secrets") or []))
miss=",".join(s.get("missing") or [])
rows.append("%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", s.get("deploymentId")))
rem=",".join(s.get("removed") or [])
rows.append("%s|%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", rem or "-", s.get("deploymentId")))
print("\n".join(rows))')
echo "$ROWS" | sed 's/^/ audit: /'
R2=$(jget "$C1" rev); R4=$(jget "$C2" rev)
EXPECTED="site-a|True|$SECRET@r$R2|-
site-a|True|$SECRET@r$R4|-
site-a|False|-|$MISSING
site-b|False|$SECRET@r$R4|-"
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-4)
[ "$GOT" = "$EXPECTED" ] && ok "exactly 4 rows, one per secret-bearing deploy (2, 4, 6-fail-closed, 7-kek-mismatch); the no-secret deploy of check 5 wrote none" \
EXPECTED="site-a|True|$SECRET@r$R2|-|-
site-a|True|$SECRET@r$R4|-|-
site-a|True|-|-|$SECRET
site-a|False|-|$MISSING|-
site-b|False|$SECRET@r$R4|-|-"
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-5)
[ "$GOT" = "$EXPECTED" ] && ok "exactly 5 rows: deploys 2 and 4, the removal-only deploy of check 6 (removed=$SECRET), 7 fail-closed, 8 kek-mismatch" \
|| { bad "audit rows differ from expected"; echo " expected:"; echo "$EXPECTED" | sed 's/^/ /'; }
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f5 | sort -u | grep -c .)" = "4" ] && ok "4 distinct deploymentIds" || bad "deploymentIds not distinct"
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f6 | sort -u | grep -c .)" = "5" ] && ok "5 distinct deploymentIds" || bad "deploymentIds not distinct"
# Material to hunt for: both plaintext values and both ciphertexts (base64, as they appear in any
# JSON serialization). Written to a pattern file so they never reach argv or the output.
PAT="$WORK/patterns"; : >"$PAT"; chmod 600 "$PAT"
@@ -584,10 +664,11 @@ done
AUDH=$(printf '%s' "$AUD" | grep -c -F -f "$PAT" || true)
[ "$HITS" = "0" ] && ok "0 hits across all 8 nodes' logs since $LOG_SINCE" || bad "$HITS log line(s) carry secret material"
[ "$AUDH" = "0" ] && ok "0 hits in the DeploySiteSecrets audit payloads" || bad "audit payload carries secret material"
end_check 8
end_check 9
echo ""
echo "=== Summary ==="
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check5=${STANDBY_5:-?}"
echo "PASS $CHECKS_PASSED/8"
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check6=${STANDBY_5:-?}"
echo "standby mismatch on central's health report (check 5): shown after ${MISMATCH_S:-?}s, cleared after ${CLEARED_S:-?}s"
echo "PASS $CHECKS_PASSED/9"
[ "$CHECKS_FAILED" -eq 0 ] || exit 1
+3 -3
View File
@@ -177,12 +177,12 @@ Options are bound via the .NET Options pattern (`IOptions<T>`). Each component o
| Key | Default | Description |
|-----|---------|-------------|
| `SeedNodes` | — | List of Akka seed-node URIs (`akka.tcp://scadabridge@host:port`). At least 2 required. Must reference remoting ports, not gRPC ports. |
| `SplitBrainResolverStrategy` | `keep-oldest` | Active strategy name (e.g. `"keep-oldest"`). |
| `StableAfter` | `"00:00:15"` | Duration the cluster must be stable before the resolver acts. |
| `SplitBrainResolverStrategy` | `auto-down` | Downing strategy: `"auto-down"` (default) or `"keep-oldest"`. |
| `StableAfter` | `"00:00:15"` | Sustained unreachability before downing (`auto-down-unreachable-after` under `auto-down`). |
| `HeartbeatInterval` | `"00:00:02"` | Akka failure-detector heartbeat cadence. |
| `FailureDetectionThreshold` | `"00:00:10"` | Acceptable heartbeat pause before a node is considered unreachable. |
| `MinNrOfMembers` | `1` | Minimum cluster members before the leader is elected. |
| `DownIfAlone` | `true` | When using `keep-oldest`, whether a lone surviving node downs itself. |
| `DownIfAlone` | `true` | `keep-oldest` only; inert under `auto-down`. |
### `ScadaBridge:Database` → `DatabaseOptions`
+2 -2
View File
@@ -20,7 +20,7 @@ Before the current design, site nodes delivered notifications directly over SMTP
This means:
- Credential exposure is limited to the central cluster.
- List membership is resolved at delivery time, so a list change takes effect for all future deliveries without redeploying to sites.
- The SMTP `MaxConcurrentConnections` value is configured at a single point, though it is not currently enforced (no connection gate or semaphore).
- The SMTP `MaxConcurrentConnections` value is configured at a single point and now clamps the Notification Outbox dispatcher's Email delivery parallelism (review 2026-10-21 06/IMPL-2); it is still unused for the EWS transport.
### `NotificationType` discriminator
@@ -156,7 +156,7 @@ services.AddNotificationOutbox(); // registers EmailNotificationDeliveryAdapte
| Section | Key | Default | Description |
|---------|-----|---------|-------------|
| `ScadaBridge:Notification` | `ConnectionTimeoutSeconds` | `30` | SMTP connection/operation timeout in seconds. Applied when `SmtpConfiguration.ConnectionTimeoutSeconds` is zero or negative. |
| `ScadaBridge:Notification` | `MaxConcurrentConnections` | `5` | Maximum concurrent SMTP connections. Used as the documented fallback default when the `SmtpConfiguration` row is unset; this limit is not currently enforced by a connection gate or semaphore. |
| `ScadaBridge:Notification` | `MaxConcurrentConnections` | `5` | Maximum concurrent SMTP connections. Used as the fallback default when the `SmtpConfiguration` row is unset; the outbox dispatcher clamps Email parallelism to it (unused under EWS). |
SMTP retry settings (`MaxRetries`, `RetryDelay`) live on the `SmtpConfiguration` entity and are read by the Notification Outbox dispatcher — they are not part of `NotificationOptions`.
+16 -23
View File
@@ -1,6 +1,6 @@
# Security & Auth
The Security & Auth component handles user authentication against an LDAP/Active Directory server and enforces role-based authorization across all central cluster operations. It owns the cookie+JWT hybrid session model, the LDAP-group-to-role mapping pipeline, site-scoped deployment permissions, and the inbound API key management seam.
The Security & Auth component handles user authentication against an LDAP/Active Directory server and enforces role-based authorization across all central cluster operations. It owns the cookie session model (interactive UI) and the bearer-JWT model (CLI/API), the LDAP-group-to-role mapping pipeline, site-scoped deployment permissions, and the inbound API key management seam.
## Overview
@@ -8,7 +8,7 @@ Security & Auth (#10) runs exclusively on the central cluster — sites have no
The component registers:
- `JwtTokenService` — token generation, validation, idle-timeout enforcement, and sliding refresh logic.
- `JwtTokenService` — mints and validates the short-lived HS256 **bearer** JWT issued by `POST /auth/token` for CLI/API callers (it no longer participates in the interactive cookie session; the sliding-refresh/idle members were removed in the 2026-10-21 review remediation).
- `RoleMapper` — DB-backed LDAP-group-to-role resolution with site-scope union semantics.
- `ScadaBridgeGroupRoleMapper` — adapter exposing `RoleMapper` on the shared `IGroupRoleMapper<string>` seam.
- `HttpAuditActorAccessor` — resolves the authenticated username from the ambient HTTP context for audit `Actor` stamping.
@@ -21,15 +21,15 @@ The component registers:
Authentication uses a direct username/password bind against the LDAP/AD server via the shared `ILdapAuthService` (`ZB.MOM.WW.Auth.Ldap`). The flow is: service-account bind → search for the user entry by username → user-credential bind → group-membership query. The app never caches credentials locally. LDAPS (port 636) or StartTLS is required for production; the `AllowInsecure` flag in `LdapOptions` gates unencrypted use to explicitly opted-in deployments (local dev only). No Kerberos/NTLM path exists.
LDAP failure behavior is fail-closed at the login boundary and fail-open at the session boundary: a new login fails immediately if the directory is unreachable; an active session (valid cookie+JWT) continues with its current claims until the JWT expires. This avoids disrupting engineers mid-work during a brief directory outage. When the directory recovers, the next token refresh re-queries groups and issues a fresh token.
LDAP failure behavior is fail-closed at the login boundary and fail-open at the session boundary: a new login fails immediately if the directory is unreachable; an active cookie session continues with its current claims (its mid-session role refresh is DB-only and never calls LDAP), and a CLI holding a bearer token keeps working until the token expires. This avoids disrupting engineers mid-work during a brief directory outage. Group-membership changes are picked up at the next login.
### Cookie+JWT hybrid session
### Sessions: cookie (UI) and bearer JWT (CLI/API)
On successful login the server mints a JWT via `JwtTokenService.GenerateToken` and writes it into an HttpOnly/Secure authentication cookie. The cookie is the transport — not a bearer header — because Blazor Server's persistent SignalR circuits do not carry `Authorization` headers on reconnect. The browser sends the cookie on every HTTP and SignalR request automatically.
**Interactive UI.** `POST /auth/login` signs the user in with ASP.NET Core cookie authentication. The HttpOnly/Secure cookie carries a data-protected claims ticket (no embedded JWT) built by the shared `SessionClaimBuilder`: display name, username, roles, permitted site IDs (site-scoped Deployer), the raw LDAP groups and the idle / role-refresh anchors. The cookie is the transport because Blazor Server's SignalR circuits do not carry `Authorization` headers on reconnect. Both central nodes share the Data Protection key ring, so either node validates a cookie the other issued. The cookie's `ExpireTimeSpan` is `IdleTimeoutMinutes` (default 30) with sliding expiration; passive traffic does not renew it. Once the role-refresh anchor is older than `RoleRefreshThresholdMinutes` (default 15) the validation hook re-runs the DB-backed `RoleMapper` on the stored groups (no LDAP call).
The JWT is signed with HMAC-SHA256 using a shared symmetric key (`JwtSigningKey`). Both central nodes share the same key, so either node can issue and validate tokens without a shared session store; the load balancer needs no sticky-session configuration. `ClockSkew` is set to `TimeSpan.Zero` to close the standard five-minute tolerance window.
**CLI / API.** `POST /auth/token` performs one LDAP bind per invocation and returns a short-lived HS256 JWT (`JwtExpiryMinutes`, default 15) signed with `JwtSigningKey`, shared by both central nodes; the response advertises `management_bearer: true`. `ManagementAuthenticator` accepts `Authorization: Bearer` alongside HTTP Basic on `POST /management`, the audit REST endpoints and the debug-stream hub. Roles inside a valid token are authoritative until it expires, so a revocation reaches a bearer caller within `JwtExpiryMinutes`; rotating `JwtSigningKey` invalidates every token. Under `DisableLogin`, `/auth/token` answers 404. Full spec: `docs/requirements/Component-Security.md`.
Claims embedded in every token:
Claims in the bearer token:
| Claim type | Constant | Value |
|---|---|---|
@@ -37,15 +37,8 @@ Claims embedded in every token:
| `ZbClaimTypes.Username` | `JwtTokenService.UsernameClaimType` | Authenticated username |
| `ClaimTypes.Role` (URI) | `JwtTokenService.RoleClaimType` | One claim per role |
| `ZbClaimTypes.ScopeId` | `JwtTokenService.SiteIdClaimType` | One claim per permitted site (Deployer only) |
| `"LastActivity"` | `JwtTokenService.LastActivityClaimType` | ISO 8601 idle-timeout anchor |
`MapOutboundClaims` is cleared on the mint path (`GenerateToken`) so `JwtSecurityTokenHandler` writes claim type strings verbatim into the token. On the validate path (`ValidateToken`) only `MapInboundClaims = false` is set — the outbound map is not touched. Together these settings prevent the default claim-type rewrite maps from transforming the canonical role URI or `zb:` claim types, keeping the type strings byte-for-byte identical in the token and in every policy check.
### Token lifecycle and idle timeout
The JWT lifetime (`JwtExpiryMinutes`, default 15 minutes) and the cookie idle window (`IdleTimeoutMinutes`, default 30 minutes) are separate layers. ASP.NET cookie auth's `SlidingExpiration = true` with `ExpireTimeSpan = IdleTimeout` models the idle window: the middleware re-issues the cookie once the session passes its halfway mark, keeping active users signed in. The JWT within that cookie has its own 15-minute expiry.
`JwtTokenService.ShouldRefresh` checks whether remaining JWT lifetime is below `JwtRefreshThresholdMinutes` (default 5 minutes). `RefreshToken` issues a fresh JWT while **preserving** the existing `LastActivity` anchor — a background refresh is not treated as user activity. `RecordActivity` advances the anchor to now. `IsIdleTimedOut` checks whether the elapsed time since `LastActivity` exceeds `IdleTimeoutMinutes`; `RefreshToken` enforces the idle check internally so an idle-expired session cannot be kept alive by background polling regardless of caller discipline (Security-014).
Claim-type maps are disabled on mint and validate (`MapInboundClaims = false`), so the canonical role URI and `zb:` claim types stay byte-identical in the token and in every policy check; validation allows a 30 s clock skew for node-to-node drift.
## Architecture
@@ -143,10 +136,9 @@ Login flow (Central UI `/auth/login` and `/auth/token`):
1. Call `ILdapAuthService.AuthenticateAsync(username, password)` (registered by Host via `AddZbLdapAuth`).
2. On success, call `RoleMapper.MapGroupsToRolesAsync(ldapGroups)` to resolve roles and site scope.
3. Call `JwtTokenService.GenerateToken(displayName, username, roles, permittedSiteIds)` to mint a signed JWT.
4. Write the JWT into the HttpOnly cookie via the ASP.NET cookie auth `SignInAsync`.
3. `/auth/login`: build the claims via `SessionClaimBuilder` and call the ASP.NET cookie `SignInAsync`. `/auth/token`: call `JwtTokenService.GenerateToken(displayName, username, roles, permittedSiteIds)` and return the bearer JWT.
On each subsequent request, middleware reads the cookie, validates the embedded JWT with `JwtTokenService.ValidateToken`, and checks `IsIdleTimedOut`. If the token is near expiry (`ShouldRefresh`), fresh claims are re-queried from LDAP and `RefreshToken` issues a replacement. Genuine user interactions call `RecordActivity` to advance the last-activity anchor.
On each subsequent cookie request the validation hook enforces the idle timeout and, past the role-refresh threshold, re-maps the stored groups against the DB. Bearer requests validate the token with `JwtTokenService.ValidateToken` and perform no LDAP bind or role re-map.
Authorization gates use the named policies:
@@ -167,10 +159,11 @@ if (!AuthorizationPolicies.OperationalAuditRoles.Contains(userRole))
| Key | Default | Description |
|---|---|---|
| `JwtSigningKey` | *(required)* | Symmetric HMAC-SHA256 signing key. Must be at least 32 bytes (256 bits); validated at `JwtTokenService` construction — startup fails if too short. |
| `JwtExpiryMinutes` | `15` | JWT lifetime in minutes before embedded token expires and must be refreshed. |
| `JwtRefreshThresholdMinutes` | `5` | Minutes before JWT expiry at which `ShouldRefresh` triggers a re-issue. |
| `IdleTimeoutMinutes` | `30` | Session idle timeout in minutes. Cookie `ExpireTimeSpan` is set to this value; `IsIdleTimedOut` enforces it from the `LastActivity` claim. |
| `JwtSigningKey` | *(required)* | Symmetric HMAC-SHA256 signing key. Must be at least 32 bytes (256 bits); `StartupValidator` rejects a shorter key on Central. |
| `JwtExpiryMinutes` | `15` | Lifetime of the bearer JWT issued by `/auth/token`. |
| `IdleTimeoutMinutes` | `30` | Cookie session idle timeout; set as the cookie `ExpireTimeSpan`. |
| `RoleRefreshThresholdMinutes` | `15` | Age of the role-refresh anchor after which the cookie's roles are re-mapped from the DB (LDAP-free). |
| `RoleMappingCacheSeconds` | `30` | Process-wide `RoleMappingCache` TTL (`0` disables); invalidated on the local node after mapping/scope-rule edits. |
| `RequireHttpsCookie` | `true` | When `true`, the cookie is `Secure`-only (HTTPS required). Set `false` for HTTP-only dev deployments; a warning is logged at startup. |
| `CookieName` | `ZB.MOM.WW.ScadaBridge.Auth` | Authentication cookie name. Override per deployment when two ScadaBridge stacks share a hostname — browsers scope cookies by host+path but not by port. |
@@ -211,7 +204,7 @@ Maps from `LdapAuthFailure.GroupLookupFailed`. The user-credential bind succeede
### Session expires unexpectedly
Check `IdleTimeoutMinutes` and `JwtExpiryMinutes` in `SecurityOptions`. A background refresh that fires while the user is idle preserves the `LastActivity` anchor (`RefreshToken` does not advance it); `IsIdleTimedOut` enforces the window from the last genuine user interaction. If the idle timeout fires before the expected window, confirm that `RecordActivity` is being called on genuine user requests.
Check `IdleTimeoutMinutes` and `JwtExpiryMinutes` in `SecurityOptions`. Passive traffic (`/auth/ping`, Blazor transport validation) does not renew the cookie; only the antiforgery-protected `/auth/activity` endpoint does. If the idle timeout fires earlier than expected, confirm genuine user interaction reaches that endpoint.
### Two ScadaBridge environments on the same host clobber each other's session
+1 -1
View File
@@ -38,7 +38,7 @@ The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGat
### Automatic failover
When the active central node goes down, the Akka cluster's keep-oldest split-brain resolver leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second stable-after period). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
When the active central node goes down, the Akka cluster's `auto-down` downing strategy (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
## Architecture
+1 -2
View File
@@ -27,7 +27,7 @@ bundle.scadabundle
`manifest.json` is always plaintext so the import wizard can display source provenance and artifact counts before the operator supplies a passphrase. `BundleManifest` carries: `BundleFormatVersion`, `SchemaVersion`, `CreatedAtUtc`, `SourceEnvironment`, `ExportedBy`, `ScadaBridgeVersion`, `ContentHash` (`sha256:<hex>` of the raw content bytes), optional `Encryption` metadata, a `Summary` (artifact counts by type), and a `Contents` list (one `ManifestContentEntry` per artifact with its `dependsOn` edges).
`BundleFormatVersion` is an integer gate: the importer requires `BundleFormatVersion` to equal `ManifestBuilder.CurrentBundleFormatVersion` (currently `1`) and rejects any other value — higher or lower — with `ManifestValidationResult.UnsupportedFormatVersion`. `TransportOptions.SchemaVersionMajor` is not read during import. Unknown entity types in `Contents` produce a preview-row classification of "unsupported" rather than aborting the whole import.
`BundleFormatVersion` is an integer gate: the importer requires `BundleFormatVersion` to equal `ManifestBuilder.CurrentBundleFormatVersion` (currently `1`) and rejects any other value — higher or lower — with `ManifestValidationResult.UnsupportedFormatVersion`. There is no configurable schema-version option; both versions are code constants. Unknown entity types in `Contents` produce a preview-row classification of "unsupported" rather than aborting the whole import.
### Encrypted vs plaintext bundles
@@ -229,7 +229,6 @@ After import, template changes propagate to deployed instances through revision-
| Key | Default | Description |
|-----|---------|-------------|
| `SourceEnvironment` | `"scadabridge"` | Environment label stamped in `manifest.json` and used in export filenames. |
| `SchemaVersionMajor` | `1` | Major schema version stamped in exported manifests. Not read by the importer; import version-gating uses `ManifestBuilder.CurrentBundleFormatVersion` directly. |
| `BundleSessionTtlMinutes` | `30` | TTL for an in-progress import session. |
| `MaxBundleSizeMb` | `100` | Upload size cap; enforced before any decompression. |
| `MaxBundleEntryCount` | `4` | Maximum ZIP entries (a valid bundle has exactly 2). |
@@ -0,0 +1,84 @@
# 2026-10-22 — DEV (wonder-app-vd03) `b3021e7c` → `3808b49f`
**Build:** `main` @ `3808b49f` (= `origin/main`), ProductVersion `1.0.0+3808b49fc7558a177219d00eac97747b28850a80`,
published from a detached worktree at that SHA (`dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true
-p:PublishSingleFile=false`; 1028 app files, 223 cli files, zip 125 543 156 B, SHA-256 `770bb992…5fb1098e47`).
**Delta:** 191 commits. Highlights:
- site secrets ride Deploy Artifacts (the gRPC secrets hub is retired);
- the 2026-10-21 system-review remediation (CLI bearer tokens, config-audit retention, actor-name gate, per-lane S&F sweep,
audit forwarding states, …);
- Topology fixes and features: area rename persists (#38), first-layer preload (#39), **Deploy all…** plus CLI `deploy area` (#40).
**QA (vq03/vq04) and prod NOT touched.**
**Transport:** the same as 2026-09-28.
- The Mac-side `winrm` CLI ran two on-box scripts (`sb-prep.ps1`, `sb-swap.ps1`) over WinRM 5985.
- Files went over SMB 445.
- Baseline, verification and logs went over HTTP and SMB, with DB reads as a direct SQL login.
## Pre-flight (breaking-change check, done before the window)
| Area | Change in delta | vd03 state | Verdict |
|---|---|---|---|
| Schema | 57 → 58: `ReviewRemediationIndexes`, which adds 3 `AuditLogEntries` indexes and 2 partition-aligned `AuditLog` indexes | 57; AuditLog 1 670 rows, AuditLogEntries 465; `ps_AuditLog_Month` present | Additive, seconds to build |
| `Secrets:Replication:Enabled` ≠ false | Now a hard `StartupValidator` failure | `false` in appsettings.json; not in the service env | OK |
| `ScadaBridge:Security:JwtSigningKey` | Must be ≥ 32 bytes when set | Env value, 64 chars | OK |
| `ScadaBridge:Database:MachineDataDb` | Option removed (it was a required key) | Still in the Central service env and appsettings | Ignored, harmless; can be removed |
| `Transport:SchemaVersionMajor`, `JwtRefreshThresholdMinutes` | Options removed | Not set | OK |
| Site deploy actor-name gate (`ActorNameRules`) | Rejects names with spaces or non-ASCII | Checked all 37 instance, script, alarm, native-source and connection names | All valid |
| Data-connection `secret:` references | Artifact deploy now resolves them centrally and fails closed | The MxGateway connection carries no reference; the artifact hash is unchanged after the deploy | OK |
| CLI auth | The new CLI exchanges credentials at `/auth/token` | The new server answers 404 under DisableLogin, so the CLI falls back to Basic | OK (verified) |
| Package changes | `Secrets.Replicator.Grpc` dropped; crypto/SignalR pins raised | n/a | OK |
**No appsettings change.** The box's three `appsettings*.json` were kept and restored, with SHA-256 matches:
- appsettings.json `A6B6CF05…`
- appsettings.Central.json `3382DD83…`
- appsettings.Site.json `3E7AD9A4…`
The service registry environment was untouched.
## Procedure (`E:\psrem-stage\3808b49f\`, log `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261022T151735.txt`)
1. **Baseline** (CLI `debug snapshot`):
- Z28061 (1), Z28062 (2) and Z28061Sim (8): Bad 270 / 270. This was **already all-Bad before the window**: the MxGateway worker had been timing out on `AddItemBulk` all day, about 2 450 such site-log lines before 15:17.
- SBTEST01 (9) and SBTEST02 (10): Good 86, 5 alarms each.
2. **`sb-prep.ps1`** (15:17:08):
- Zip hash re-verified and unpacked; staged Host and CLI are `3808b49f`.
- `reg export` of both service keys.
- `BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION` → `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-3808b49f_20261022T151708.bak` (130 453 504 B); `VERIFYONLY` valid.
3. **`sb-swap.ps1`**:
- 15:17:36: Site, then Central stopped (10 s each).
- 15:17:57: `ReviewRemediationIndexes.sql` applied with `sqlcmd -b -I`; migrations 57 → 58.
- `app`/`cli` renamed to `*.bak.20261022T151735`; staged copy in; appsettings restored.
- Central ready at 15:18:16; Site ready at **15:18:24**.
- **About 49 s of downtime.**
- SQL Server warned that `IX_AuditLog_Kind_Occurred` has a 8 008-byte maximum key (the `Kind` computed column is wide). `Kind` holds short enum names, so this never bites in practice.
4. **`deploy artifacts --site-id 2`**: success.
- `artifactsDeployedAt 2026-10-22T19:20:28Z`.
- Hash unchanged (`sha256:9e054ef0…`), as designed for a site with no secret references.
## Verification
| Check | Result |
|---|---|
| `:8085` and `:8084` `/health/ready` | 200 / 200 |
| ProductVersion | Live Host and CLI are `3808b49f` |
| DB | 58 migrations |
| `POST /auth/token` | 404 (DisableLogin); the new CLI falls back to Basic in the default token mode |
| Snapshots after | Identical to the baseline |
| Logs after boot (pulled over SMB) | **Central:** the usual first-probe `akka-cluster … Removed` ERR lines plus two expected Warnings (below). **Site:** only the pre-existing MxGateway `AddItemBulk` deadline / quality-Bad lines |
| `docker/verify-mes-test-machine.sh` (with the `RECIPE_ROOT` / `RECIPE_OUTSIDE` for vd03) | **PASS 31/31** |
The two expected Central Warnings both come from the placeholder second seed. That entry makes the single-node central count as clustered:
- inbound API keys are on SQLite;
- **new:** secrets use a per-node SQLite store.
## Rollback
1. Stop Site, then Central.
2. Rename `app` → `app.failed` and `app.bak.20261022T151735` → `app`; do the same for `cli`.
3. Start Central, then Site.
The new indexes are additive and `b3021e7c` ignores the extra migration row, so no DB restore is needed. To make the DB exact anyway, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-3808b49f_20261022T151708.bak' WITH REPLACE`.
Staging is left under `E:\psrem-stage\3808b49f\`. The `_appsettings-keep-20261022T151735` and `*.bak.20261022T151735` folders can go once the build has soaked.
@@ -0,0 +1,154 @@
# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-10-22 — `b3021e7c` → `3808b49f`
This record covers the **QA two-host topology**:
| Box | Central node | Site-a node |
|---|---|---|
| vq03 `10.220.158.2` | central-a | node-a |
| vq04 `10.220.158.3` | central-b | node-b |
- Centrals on :8085; site gRPC on :8083, health on :8084.
- LocalDb replication is on.
- The shared config DB `ScadaBridge` is on `wonder-sql-vq03`.
The package is the one DEV received the same day. See `2026-10-22-wonder-app-vd03-3808b49f.md` for the build delta: 191 commits, including site secrets via Deploy Artifacts, the 2026-10-21 review remediation, and Topology #38/#39/#40.
The deploy ran from the Mac with the `winrm` CLI. The zip and SQL went over SMB, followed by one prep script per box and seven window calls, all from one mode-switched script.
## Pre-flight (breaking-change check)
| Area | Change in delta | QA state | Verdict |
|---|---|---|---|
| Schema | 57 → 58 `ReviewRemediationIndexes` (5 additive indexes) | 57; AuditLog 2 132 rows, AuditLogEntries 515; `ps_AuditLog_Month` present; Enterprise | OK |
| `Secrets:Replication:Enabled` ≠ false → startup failure | — | `false` in appsettings.json on both boxes; not in the service env | OK (stale `Mode`/`GrpcHub` keys log one ignore-Warning) |
| `JwtSigningKey` ≥ 32 bytes; must match on both centrals now that the CLI uses bearer tokens | — | 64 chars on both; identical hash | OK |
| `MachineDataDb`, `Transport:SchemaVersionMajor`, `JwtRefreshThresholdMinutes` removed | — | `MachineDataDb` still in the Central env and appsettings; the others are unset | Ignored, harmless |
| Actor-name deploy gate | — | 33 instance, script, alarm, native-source and connection names | All valid |
| LocalDb replicated set 10 → 12 tables (`deployed_secrets`, `secret_mirror_status`) | Registered-table digests are compared fail-closed | Both site nodes upgraded in one window | OK; the pair reconnected, backlog 0 |
| Data-connection `secret:` references | Artifact deploy resolves them centrally | The artifact hash is unchanged after the deploy, so there are no references | OK |
| Service env | — | Re-exported during prep; identical to the 2026-09-28 export on all four services | OK |
**No appsettings change.** The three files were kept and restored on each box, with matching hashes:
| Box | appsettings.json | appsettings.Central.json | appsettings.Site.json |
|---|---|---|---|
| vq03 | `F94000B1` | `D1320FA8` | `4CBAF66C` |
| vq04 | `F94000B1` | `2FD3EC4C` | `D0C1AF05` |
## Upgrade
| Item | Value |
|---|---|
| Build | `main` @ **`3808b49f`**, ProductVersion `1.0.0+3808b49fc7558a177219d00eac97747b28850a80`, all four nodes |
| Artifact | `E:\psrem-stage\3808b49f\scadabridge.zip` on both boxes; 125 543 156 B; SHA-256 `770BB992…5FB1098E47`, re-verified on each box; app 1028 files, cli 223 |
| Migration | `ReviewRemediationIndexes.sql` (idempotent) applied with `sqlcmd -C -b -I` from vq03 while both centrals were stopped; 57 → 58. SQL Server warned that `IX_AuditLog_Kind_Occurred` has a maximum key of 8 008 bytes against the 1 700-byte limit. `Kind` holds short enum names, so this is benign |
| DB backup | `F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak` on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; VERIFYONLY valid) |
| Binary backups | `app.bak.20261022T152457` / `cli.bak.20261022T152457` plus `_appsettings-keep-20261022T152457` on both boxes |
| Notes on box | `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-3808b49f.txt` |
| Step | Local time |
|---|---|
| Stop Site vq04 (**window opens**) | 15:29:58 → 15:30:28 |
| Stop Site vq03 | 15:30:37 → 15:30:47 |
| Stop Central vq04, swap vq04 | 15:30:55 → 15:31:12 |
| Stop Central vq03, migrate, swap vq03, start Central vq03 → ready | 15:31:19 → 15:31:49 |
| Start Central vq04 → ready | 15:32:00 → 15:32:05 |
| Start Site vq03 → ready | 15:32:12 → 15:32:21 |
| Start Site vq04 → ready (**window closes**) | 15:32:30 → 15:32:33 |
**Downtime: 2 min 35 s.**
## Verification (from the Mac)
- **Centrals:** `/health/ready` 200 on both. `database`, `akka-cluster` (memberCount 2, unreachable 0) and `required-singletons` are all Healthy. `/health/active` is 200 on vq03 and 503 on vq04, the same active node as before.
- **Sites:** `/health/ready` 200 on both. `localdb` shows `replicationConnected:true`, backlog 0. `/health/active` is 200 on vq03 and 503 on vq04.
- **Auth:**
- `POST /auth/token` returns 200; the CLI now runs on bearer tokens against QA.
- `/management` without credentials returns 401, unchanged.
- **Snapshots:**
- Z28061–Z28069: 242 Good / 2 Bad on both sides of the window. Z28063 showed 114 Uncertain on the first read while its subscriptions settled, then matched.
- Alarm counts are unchanged (14, or 8 on Z28062).
- ZIPSEN is not deployed, as before.
- **Deploy Artifacts:** deployment `d7505cd8…`, site-a success. The hash is unchanged (`sha256:d79ba1a3…`); `ArtifactsDeployedAt` is 19:32:56Z.
- **Logs** (pulled over SMB):
- The usual first-probe `akka-cluster … Removed` ERR lines.
- vq03 site logged four `Replication session … faulted` Warnings between 15:32:16 and 15:32:29, while vq04's site was still down. The session has been connected since.
- New startup Warnings: Secrets per-node SQLite on a clustered central, and the stale `Secrets:Replication`/`GrpcHub` keys.
- No other ERR or FTL lines.
- **Smoke:** `flow-debug run-method` of `SimpleAlarmStatusRequest` (id 9) with `SAPID=000000` returned InboundRequest → BTDB DbCall (1 row) → Response Ok.
## Rollback
1. Stop Site vq04 → vq03, then Central vq04 → vq03.
2. On each box rename `app` → `app.failed` and `app.bak.20261022T152457` → `app`; do the same for `cli`.
3. Start Central vq03 → vq04, then Site vq03 → vq04.
The indexes are additive, so `b3021e7c` runs against 58 migrations. For an exact DB, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE` with everything stopped.
## Follow-up, same day: shared key and secret stores on the central pair (15:45–15:50)
**Pre-check (SMB copies of the per-node files):**
- Inbound keys: vq03 held 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`); vq04 held **none**. Before this change, a failover to vq04 would 401 every inbound client.
- Secrets: both central SQLite stores held **0** secrets, so there was nothing to migrate (the secrets store has no automatic importer).
- SQL login: `db_owner` on `ScadaBridge`.
**Change:**
| Where | Change | How |
|---|---|---|
| `appsettings.Central.json` (both boxes) | `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` (connection string defaults to the ConfigurationDb) and `Secrets:Store:Provider=SqlServer`, plus a `_comment_SharedStores` note | Edited on the Mac, pushed over SMB while Central was stopped |
| `appsettings.json` (both boxes) | Removed the stale `Secrets:Replication` (incl. `Mode`) and `Secrets:GrpcHub` blocks and the dead `SqlServer` sync keys | Edited on the Mac, pushed over SMB while Central was stopped |
| Central service registry env (both) | Added `Secrets__SqlServer__ConnectionString` | Copied on the box from `ScadaBridge__Database__ConfigurationDb`; the value never left the box. Site services untouched |
Backups: `E:\psrem-stage\3808b49f\_envbak-ScadaBridge-Central-pre-sharedstores-<ts>.reg` and `E:\ApiInstall\ScadaBridge\_appsettings-keep-sharedstores-<ts>`, on each box.
**Sequence:**
1. Both centrals stopped together (15:45:42 / 15:45:55).
2. vq03 started → ready 15:46:26; vq04 started → ready 15:46:39. The `zbauth` and `zbsecrets` schemas were created in `ScadaBridge`.
3. vq03 imported **3 keys + 3 audit rows**; vq04 imported 0.
**Defect found and fixed:** on both nodes the import's rename to `*.imported` failed with `IOException: … being used by another process`.
- Cause: the library importer reads the file through a pooled `SqliteConnection`, whose handle outlives the connection. Windows refuses to rename an open file (Linux doesn't mind, so the rig never showed it).
- Reproduced on the Windows build box: the rename fails without `SqliteConnection.ClearAllPools()` and succeeds with it.
- Code fix: `6ed6800b`, which clears the pools before the rename, plus a regression test and a runbook note.
- On QA the files were renamed by hand with each Central stopped:
- vq04 first. Its first attempt hit the same lock because its Host process was still exiting, and the script stopped before restarting it, so **vq04's Central was down 15:48:55–15:50:24**.
- vq03 at 15:49:07–15:49:28.
- These two windows overlapped, so **the central API was unavailable for about 20 s**. The sites were unaffected.
**Verified:**
- Both centrals ready; 2 members, 0 unreachable; vq03 active.
- `security api-key list` returns the same 3 keys on **both** nodes.
- An inbound call with a bogus key gets 401 `UNAUTHORIZED` on the active node.
- No startup warnings after the last start (the Secrets-SQLite, inbound-key-SQLite and stale-key warnings are gone).
- `central\data` on each box now holds `inbound-api-keys.sqlite{,-wal,-shm}.imported` and the (empty) `secrets.db`.
**Rollback:**
1. Restore the two appsettings files from `_appsettings-keep-sharedstores-<ts>` and import the `.reg` backup, or remove the new env value.
2. Rename the `*.imported` files back.
3. Restart both centrals.
The `zbauth` and `zbsecrets` rows can stay; the SQLite path never reads them.
## Follow-up 2: inbound API standby relay (16:57)
`ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` was added to each box's `appsettings.Central.json`:
- vq03 → `["http://10.220.158.3:8085"]`
- vq04 → `["http://10.220.158.2:8085"]`
`StandbyForwardTimeout` stays at its 60 s default, which is above QA's longest method timeout (`MesMoveIn` 46 s). The relay signature uses `ApiKeyPepper`, which is identical on both centrals.
**Rolling restart, standby first:**
1. vq04: stop 16:57:09, service process exited, file pushed over SMB, ready 16:57:28.
2. Waited for 2 members / 0 unreachable.
3. vq03: stop 16:57:33, ready 16:57:51.
The cluster never dropped below one serving central. **vq04 is now the active node** (failover on vq03's restart); nothing depends on which node is active.
Backups: `E:\ApiInstall\ScadaBridge\_appsettings.Central.json.pre-relay-<ts>` on each box.
**Verified:** a bogus-key inbound call answers **401 `UNAUTHORIZED` on both nodes**. The standby (vq03) used to answer 503 `STANDBY_NODE`; it now relays to the active peer, and the peer's answer is replayed.
## Outstanding
- **Housekeeping after soak:** delete the `*.imported` key files; also the older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.
+1 -2
View File
@@ -58,8 +58,7 @@ C:\ScadaBridge\
]
},
"Database": {
"ConfigurationDb": "Server=sqlserver.example.com;Database=ScadaBridge;User Id=scadabridge_svc;Password=<CHANGE_ME>;Encrypt=true;TrustServerCertificate=false",
"MachineDataDb": "Server=sqlserver.example.com;Database=ScadaBridge_MachineData;User Id=scadabridge_svc;Password=<CHANGE_ME>;Encrypt=true;TrustServerCertificate=false"
"ConfigurationDb": "Server=sqlserver.example.com;Database=ScadaBridge;User Id=scadabridge_svc;Password=<CHANGE_ME>;Encrypt=true;TrustServerCertificate=false"
},
"Security": {
"LdapServer": "ldap.example.com",
-1
View File
@@ -13,7 +13,6 @@
### Central Node
- [ ] `ScadaBridge:Database:ConfigurationDb` connection string is valid and tested
- [ ] `ScadaBridge:Database:MachineDataDb` connection string is valid and tested
- [ ] SQL Server login has `db_owner` role on both databases
- [ ] EF Core migrations have been applied (SQL script reviewed and executed)
- [ ] `ScadaBridge:Security:JwtSigningKey` is at least 32 characters, randomly generated
+7 -7
View File
@@ -8,16 +8,16 @@
1. Active central node becomes unreachable (process crash, network failure, hardware failure).
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
3. Split-brain resolver (keep-oldest) evaluates cluster state for 15 seconds (stable-after).
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
4. Standby node is promoted to active. Total time: ~25 seconds.
5. Cluster singletons migrate to the new active node.
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
7. Active user sessions continue (JWT tokens are validated by the new node using the shared signing key).
7. Active user sessions continue (the cookie's data-protected ticket is validated by the new node via the shared Data Protection key ring; CLI bearer tokens via the shared signing key).
8. SignalR connections are dropped and Blazor clients automatically reconnect.
**What is preserved:**
- All configuration and deployment state (stored in SQL Server)
- Active JWT sessions (shared signing key)
- Active sessions (shared Data Protection keys for cookies, shared signing key for CLI bearer tokens)
- Deployment status records (SQL Server with optimistic concurrency)
**What is temporarily disrupted:**
@@ -82,9 +82,9 @@
**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".
**How the system handles it:**
- Keep-oldest resolver: The older node (first to join cluster) survives; the younger is downed.
- `down-if-alone = on`: If a node is alone (no peers), it downs itself.
- Stable-after (15s): The resolver waits 15 seconds for the partition to stabilize before acting.
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
- Stable-after (15s): the window of sustained unreachability before the downing decision.
**Manual intervention (if auto-resolution fails):**
1. Stop both nodes.
@@ -96,7 +96,7 @@
**Symptoms:** Central UI returns errors. `/health/ready` returns 503. Logs show database connection failures.
**Impact:**
- Active sessions with valid JWTs can still access cached UI state.
- Active sessions (valid cookie or bearer token) can still access cached UI state.
- New logins fail (LDAP auth still works but role mapping requires DB).
- Template changes and deployments fail.
- Sites continue operating independently.
@@ -71,6 +71,12 @@ about this at boot). To move an existing pair over:
5. Check the import log line on each node (`grep -i "api-key store import\|zbauth"` in the
node's log) — one node reports the keys it inserted, the other reports the ones it found
already present (skipped); together they cover the union of both nodes' prior SQLite files.
On a **Windows** central running a build before 2026-10-22 the import succeeds but the
`*.imported` rename then fails with `IOException: … being used by another process` (the
importer's pooled SQLite handle; fixed by clearing the pools before the rename) — the import
re-runs harmlessly on every boot. Fix it by hand: stop that node's Central, wait for its
`ZB.MOM.WW.ScadaBridge.Host` process to exit, rename `inbound-api-keys.sqlite` and its
`-wal`/`-shm` siblings to `*.imported`, start it again (QA, 2026-10-22).
6. Run `security api-key list` against each node's management endpoint and confirm both show
the same keys.
7. After a quarantine period (once you're confident the import is correct and no rollback is
@@ -29,7 +29,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 33 | ~~**`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment**~~ **RESOLVED 2026-08-15 (`8da60f57`).** Both halves shipped: sampled into KPI history via `SiteHealthKpiSampleSource` (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (`LocalDbBacklogThreshold` 1000 held for `LocalDbBacklogSustainedFor` 15 min, anchored by `LocalDbBacklogAboveSince` in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the `SiteAuditBacklog` "configurable threshold badge" it told us to mirror was described twice in `Component-HealthMonitoring.md` but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. | Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| 34 | ~~**`site_events` purge burst is unbounded in the oplog and unmeasured at scale**~~ **RESOLVED 2026-08-16 (`d9a35846`) — measured, and paced.** Measured with the REAL `EventLogPurgeService` over a real `RegisterReplicated` LocalDb file + a loopback sync pair at the rig's `MaxBatchSize: 64`: 393 B/row incl. indexes ⇒ a 1 GB event log holds **~2.73 M rows**, and expiring them in one tick took 285 s and queued **2 731 947 oplog rows = 2.7x LocalDb's `MaxOplogRows` default (1 M)** — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the *resident* backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: `MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (**5 min**, validated `> 0` and `<= PurgeInterval`) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter `scadabridge.site.event_log.purge_ceiling_hits`. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: `PurgeBudget` neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). | Found by R7 | Closed 2026-08-16. | — |
| 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* **2026-10-21 note:** the same leading-wildcard `LIKE` shape exists on Instances and `AuditLogEntries` free-text search. `ListInstances` is no longer one of them — it pages, site-scopes and **prefix**-searches in SQL as of the 2026-10-21 review remediation (`baab6ac7`; CLI `instance list --search` is now a prefix match) — but the config-audit viewer filters and the Deployments page remain scans. | Search latency complaints on the Deployments page or the config audit log |
| 37 | ~~**Two paged reads of `DeploymentRecords` with different contracts**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** `QueryDeploymentSummariesAsync` and `DeploymentRecordSummary` are DELETED; the CLI/HTTP `QueryDeployments` handler now rides `QueryDeploymentListPageAsync` + `DeploymentListFilter` — one row shape (`DeploymentListRow`), one scoping mechanism (site-id via the query join), total count available to both surfaces (`DeploymentListPage` now also owns `MaxPageSize` + `DefaultPageSize = 50`). Rows 86–88 were resolved with the consolidation, as this row anticipated. Row 36's 4-case substring-search Theory untouched and still passing. Remaining UI-vs-CLI asymmetry (`GetDeploymentStatusCountsAsync` status tiles are UI-only) is deliberate, not drift. **Live-probed on the rig:** CLI `deploy status --status Success --page-size 5` returns the consolidated page — rows now carry `siteId` + `instanceUniqueName`, and the response carries `totalCount` (11) the CLI never had. | Found by R3; consolidation shipped with rows 86–88 | Closed 2026-08-17. | — |
| 38 | ~~**OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile**~~ **RESOLVED 2026-08-18 in OtOpcUa (`34829ea0` + `837d8a17`, branch perf/followups-r3).** Rebuilt on the assert-on-retry-exhaustion option, renamed `…never_leaks_a_raw_disposed_failure`. Two findings beyond the brief: (1) the DOMINANT load-failure mode was the test's own 45 s `WaitAsync` budget — every clear forces a recompile, so the fixed 12×400 workload's wall-clock tracked spare CPU; reproduced RED 4/4 at 4-way self-concurrency, fixed by duration-bounding the loops (10 s window OR 400 iters), GREEN 4/4 and 8/8 after. (2) A second flake vector: `TimedScriptEvaluator`'s 5 s clock starts at enqueue, so pool starvation surfaces "script timed out" from healthy evaluations — now its own bounded bucket, not counted as a leak; failure-reason literals promoted to shared consts. Honest caveat recorded: the disposed race itself does not reproduce on the arm64 macOS box (zero ODEs even with the product retry disabled), so the raw-escape assertion is the load-independent guarantee. Commit stats cross-checked from this repo. | Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
| 39 | ~~**OtOpcUa: late-enabled replication cannot baseline pre-existing rows**~~ **RESOLVED 2026-08-18 in OtOpcUa (`3e62f161` + review round `df533916`, branch perf/followups-r3).** ScadaBridge's WP1.3/WP3.3 pattern ported: registration follows replication config — configured ⇒ `RegisterReplicated(..., baselineExistingRows: true)` (probed live on 0.2.1: seeds `__localdb_row_version` at HLC 0), unconfigured ⇒ boot-time `DeregisterReplicated` self-heal, pair-symmetric; review round added deregister-preserves-data + partial-flip-fails-loudly tests. Green: `LocalDbSetupTests` 15/15, Host.IntegrationTests 226 pass/0 fail. **Stayed pinned to LocalDb 0.2.1 by deliberate choice** — the 0.2.1 API already carries both needed methods, and the v2→v3 one-way bookkeeping upgrade makes a 0.3.x bump a both-nodes-together operation; recorded caveat in OtOpcUa's CLAUDE.md: on 0.2.1 the snapshot-resync flag is per-DATABASE, so a late enable snapshots the whole file rather than per-table (0.3.1 available; bump logged as a deliberate future change in OtOpcUa's own follow-up register). Commit stats cross-checked from this repo. | Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
@@ -109,7 +109,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 114 | *Never issued.* The 2026-09-09 step-logs close-out minted two rows numbered **113** — this one and the "other actors' log lines" follow-up. The duplicates were renumbered **115** / **116** at final review (I2) and 114 was left unused rather than reshuffling every number after it; register numbers are permanent identifiers, cited from code comments and `CLAUDE.md`. | — | — | — |
| 115 | **Other actors' log lines are not captured for an execution — only the script's own `Log.*` and the runtime lines inside a scoped helper** *(Low)*. The Flow Debugger's step logs (2026-09-09) capture what the script logs and what `ScriptRuntimeContext`'s own logger emits while a run is traced, because those two go through one seam. What the *Instance Actor*, the *DCL*, the alarm actors or the S&F engine log while working on the same execution never reaches the graph: there are **zero** `LogContext.PushProperty` calls in the repo, `Enrich.FromLogContext()` is not configured on the Serilog pipeline (`TraceContextEnricher` stamps W3C ids only), and `site_events` has no execution-id column. Capturing them is a separate enrichment initiative: push `ExecutionId` (and instance/script) onto the ambient `LogContext` at run start, enable `Enrich.FromLogContext()`, add an execution-id column + index to `site_events`, then either sink matching entries into `ScriptLog` rows or query the site event log by execution id. | Step-logs design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` §2 + §9 ("Out of scope") | Out of proportion to the ask: it changes the logging pipeline of every actor on a site node, adds a replicated-table column (`site_events` is one of LocalDb's ten `ReplicatedTables`, so the change is a both-nodes-together schema edit), and risks a volume increase on untraced runs unless the enrichment is itself gated. The script-authored lines cover the debugging case the Flow Debugger exists for. | First investigation where the answer sits in what the Instance Actor or the DCL logged during a traced run and the operator cannot correlate it by hand. |
| 116 | **`CompileSurfaceParityTests.AccessorMirrorPairs` cannot guard the `(ScriptLogger, CompileLog)` pair — the theory has no per-member exclusion mechanism** *(Low)*. The arity theory asserts the compile mirror exposes at least as many overloads as the runtime accessor for **every** public instance method, and `ScriptLogger` has one the mirror must NOT have: `FlushDropped()`, the run-end closing-row flush, which is called by `ScriptRunLauncher`, not by a script. Adding the pair today would fail the theory for a member that is correctly absent, so the pair is left out and `Log`'s four methods are guarded only by the top-level name-superset check plus the hand-written surface tests. The fix is an exclusion seam on the theory — an optional `IReadOnlySet<string> ignoredRuntimeMethods` per row (or an `[NotScriptReachable]` marker attribute on `FlushDropped`, which also documents the intent at the declaration) — after which the pair joins `AccessorMirrorPairs`. | Found while wiring Task 15 of `docs/plans/2026-09-09-flow-debugger-step-logs.md`; test `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/CompileSurfaceParityTests.cs` | No drift is possible without a code change to `ScriptLogger`, and a missing `Log` method on the mirror surfaces immediately as a `CS1061` at the design-time gate for any script that calls it. The seam is a test-infrastructure change that touches every existing row's signature. | The next public method added to `ScriptLogger` (or to any runtime accessor whose mirror must deliberately omit a member) — at that point build the exclusion seam and add the pair. |
| 117 | **Flow Debugger site trace pull: `FlowDebugRunner.WriteAndPollAsync` fetches ONE bounded page with a fixed `since`** *(Low)*. The write-and-poll loop asks the site for `FlowDebugRunner.PullTraceEventsMax` rows per poll, always `since` the run start, and the site serves `ORDER BY OccurredAtUtc ASC LIMIT` — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 `ScriptLog` rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's `ScriptRun` row past the end of it, at which point the loop never sees the awaited run and reports a false `NO_TRIGGER_WITHIN_WINDOW` for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, `SiteStreamGrpcServer.MaxTraceEventsPerPull`), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance `since` to the last-seen `OccurredAtUtc` between polls (keeping the already-known ids so nothing is double-counted) — or to exclude `ScriptLog` from the awaited-run poll entirely and pull the log rows once, after the run has settled. | Final review of `docs/plans/2026-09-09-flow-debugger-step-logs.md` (I1); `src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs` `WriteAndPollAsync` | 2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first `NO_TRIGGER_WITHIN_WINDOW` reported for a run whose Replay of the same root DOES show the spawned `ScriptRun`. |
| 117 | ~~**Flow Debugger site trace pull: `FlowDebugRunner.WriteAndPollAsync` fetches ONE bounded page with a fixed `since`**~~ **RESOLVED 2026-09-09 (`9ee402bb`, arch-review 2026-09-09 MG-6; closed in this register 2026-10-21 — found stale by the 2026-10-21 system review).** `WriteAndPollAsync` now advances a `(SinceUtc, AfterEventId)` keyset cursor between polls and only treats a page as progress when the cursor moved, so a chatty run can no longer push a spawned run past the page end. Original description kept for traceability: *(Low)*. The write-and-poll loop asks the site for `FlowDebugRunner.PullTraceEventsMax` rows per poll, always `since` the run start, and the site serves `ORDER BY OccurredAtUtc ASC LIMIT` — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 `ScriptLog` rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's `ScriptRun` row past the end of it, at which point the loop never sees the awaited run and reports a false `NO_TRIGGER_WITHIN_WINDOW` for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, `SiteStreamGrpcServer.MaxTraceEventsPerPull`), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance `since` to the last-seen `OccurredAtUtc` between polls (keeping the already-known ids so nothing is double-counted) — or to exclude `ScriptLog` from the awaited-run poll entirely and pull the log rows once, after the run has settled. | Final review of `docs/plans/2026-09-09-flow-debugger-step-logs.md` (I1); `src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs` `WriteAndPollAsync` | 2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first `NO_TRIGGER_WITHIN_WINDOW` reported for a run whose Replay of the same root DOES show the spawned `ScriptRun`. |
| 118 | **Pre-existing bUnit flake: `CentralUI.Tests.Pages.NotificationReportDetailModalTests.CloseButton_DismissesModal` (~10 %)** *(Low)*. Fails intermittently with `UnknownEventHandlerIdException` — the test resolves the close button once and clicks it after a re-render has already retired that element's handler id. Not caused by the step-logs work: the file is untouched since `40944a0d`. The fix is a one-liner — re-`Find` the button immediately before the click (the pattern the other modal tests already use) rather than reusing an element captured earlier in the test. | Observed during the step-logs full-suite runs, 2026-09-09 | It is a test-only race with a known cause and a known fix; it was out of scope for a remediation pass whose rule was to touch only the files under review. | Next time `CentralUI.Tests` is worked on, or the first time the flake blocks a gate run. |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
| 119 | **Port the QA/prod MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `<X>CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "<X>Flag", true, "<X>CompleteFlag", true, 25 s, "<X>CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for <X>CompleteFlag to reset" : "Timeout waiting for <X> to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/prod databases. | Next QA deployment of a build ≥ this change (write a deployment record; then prod). |
@@ -120,7 +120,15 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
| 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (merged to `main` @ `23a139a4` 2026-09-28, plus the same-day follow-ups: per-site secret-store outage handling, removal audit rows, per-node mirror status on the health report, artifact-deploy CLI exit codes; rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot (after the first deploy a mismatch on either node — the standby included — shows as a *Secret mirror failing* badge on that site's Health card); (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; the upgrade also adds two replicated site tables (`deployed_secrets`, `secret_mirror_status`), so stop and start each site pair together as usual; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
| 128 | **No TLS production profile: Traefik, Akka remoting and the gRPC h2c listeners are all plaintext** *(Medium for a non-lab network; accepted for the lab)*. Only the gRPC command/control and stream boundaries carry per-site PSK auth; nothing is encrypted and Akka remoting is unauthenticated (see CLAUDE.md *Security*). A deployable TLS profile — Traefik entrypoint certs, Akka `enable-ssl` + secure cookie, gRPC over h2/TLS with the PSK retained — has never been designed end to end. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (carried from prior reviews) | Trusted-network lab/plant deployment is the stated assumption; TLS is purely additive to the PSK design and needs no protocol change, but needs cert distribution for 8+ nodes. | First non-lab deployment (any network not wholly inside the plant security boundary). |
| 129 | **`ExternalDefinitionLookupCache` is not invalidated on external-system / DB-connection edits** *(Low)*. The central short-TTL definition cache (review 05/P2) is never `InvalidateAll()`'d by the management edit paths because ManagementService has no reference to ExternalSystemGateway; the node that took the edit and its peer both see it only after `DefinitionCacheTtl` (default 10 s). | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane M) | A 10 s staleness window on a rarely-edited definition is harmless; wiring needs a new project reference or a bus event. | A report of an edit visibly not taking effect for an inbound-method call, or any increase of `DefinitionCacheTtl`. |
| 130 | **`TemplateService` member add/update still loads the whole template analysis graph** *(Low)*. 04/P6 is only partly done: the read paths use lean projections, but member add/update still materialises the full analysis graph because no repository method loads just a template's inheritance chain + composition closure. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lanes F/G, 04/P6 partial) | Designer-rate operation; the new chain-only query is a repository + service refactor with its own correctness risk. | Template-edit latency complaints on a large library, or the library growing past a few hundred templates. |
| 131 | **`SiteStreamGrpcClientFactory` never evicts idle `(siteId, endpoint)` clients** *(Low)*. Clients are keyed per site and endpoint and live until disposed with their session/factory; an endpoint that is removed or renamed in the site's address pair leaves its client (and channel) resident. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane E item 6) | Bounded by the number of address edits over a process lifetime; a rename is rare. | Sites whose gRPC addresses change often, or a channel/handle leak seen in a long-running central. |
| 132 | **Site compile-cache eviction counter is not exported as a Meter** *(Low)*. `SiteScriptCompileCache` counts evictions and logs a rate-limited Warning (review 01/SR-M2) but the counter is not a `System.Diagnostics.Metrics` instrument, and `ZbTelemetryOptions.Meters` is an allowlist (`SiteServiceRegistration.ObservedMeters`), so exporting it also needs the Host allowlist entry. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane B) | The Warning log already surfaces thrash; a Meter is observability polish. | Wanting to alert on compile-cache thrash from Prometheus. |
| 133 | **Dedicated health field for site audit rejected rows** *(Low)*. Permanent push rejections (`IngestAck.rejected_event_ids`) are counted only by the `scadabridge.site.audit.rejected_rows` counter and the `Rejected` `ForwardState`; there is no `SiteHealthReport` field/dashboard badge. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Rejections only come from row-intrinsic SQL errors and should be near-zero; the counter is sufficient for now. | The first non-zero `rejected_rows` seen in production, or an operator asking for it on the Health dashboard. |
| 134 | **An oversize `AuditLog` row (larger than the gRPC message cap) wedges both push and pull** *(Low, accepted)*. A single row bigger than the 4 MB gRPC cap cannot be shipped by either the telemetry push or the reconciliation pull, so the queue head stalls. Unreachable at the default caps (8 KB / 64 KB payload caps, `InboundMaxBytes` 1 MiB) — it needs an operator to raise caps past the transport limit. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Defaults keep every row far below the limit; guarding means a skip/park state for unshippable rows. | Any deployment that raises `InboundMaxBytes` or the payload caps toward 4 MB. |
| 135 | **Config-audit single-column indexes are possibly redundant** *(Low)*. `ReviewRemediationIndexes` added composite `AuditLogEntries` indexes `(EntityType, Timestamp DESC)`, `(User, Timestamp DESC)` and `(Timestamp DESC, Id DESC)`; the older single-column indexes on timestamp/user/entity type may now be dead weight on a write-heavy table. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane P / O) | Dropping needs a measured before/after on a populated table, not an assumption. | Measure index usage (`sys.dm_db_index_usage_stats`) after the migration has run in QA/prod for a few weeks, then drop the unused ones. |
## Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
@@ -215,11 +215,10 @@ accurate as the spec:
Grpc mode) — without that fix a central's boot-time `${secret:}` tokens and its running
`ISecretResolver` could resolve a token differently.
- **`ZB.MOM.WW.Secrets` 0.6.3 (hub sweep logging, §3.7's third bullet — `EventId`s 4/5/6,
`ISecretsHubSyncStatus`) was implemented and tested on a `scadaproj` branch but is NOT
published to the package feed, and ScadaBridge does not consume it.** Publishing is an outward
side effect that needs explicit user consent, and it would be a dead pin here regardless —
hub retirement (above) removes ScadaBridge's only dependency on the hub client entirely. It
remains available for the library's other hub consumers once published.
`ISecretsHubSyncStatus`) is published to the Gitea feed (2026-09-28, after merge), but
ScadaBridge does not consume it** — hub retirement (above) removes ScadaBridge's only dependency
on the hub client, so it stays on 0.6.2. It is there for the library's other hub consumers
(OtOpcUa).
- **Empty, not null, when a site references no secrets (Task 13 live-gate fix).** The command
builder first shipped `Secrets = null` for a site with no references ("null-when-empty", borrowed
from the hash convention). But the site treats `null` as "central shipped no secrets" and touches
@@ -237,11 +236,45 @@ accurate as the spec:
next (re)connect (no forced reconnect).
- *One `secret:` parser* (`SecretReferenceScanner.TryParseSecretReference`) shared by the scanner
and `MxGatewayDataConnection`, so a padded/case-variant reference resolves the shipped name.
- *Audit shape (§3.7 deviation).* The `DeploySiteSecrets` row carries `secrets` and `missing` —
there is **no `removed` field** — and a removal-only deploy (empty list shipped) writes **no
central row**; removals are logged on the site by `DeployedSecretsMirror`.
- *Audit shape (§3.7).* The `DeploySiteSecrets` row carries `secrets`, `missing` and `removed`.
(Shipped first without `removed`, so a removal-only deploy wrote no central row; closed by the
follow-ups below.)
- *Deployment prerequisite:* a site pair using deployed secrets must run LocalDb replication
(topology guide), since on an unreplicated pair only the node active at deploy time gets them.
- **Follow-ups closed after merge (2026-09-28).**
- *Secret-store outage is per site (review M5).* `SiteSecretsCollector` returns a store read
failure as `SiteSecretsCollection.StoreError` instead of throwing. A deploy fails only the
sites that reference a secret (nothing is sent to them; they get a `DeploySiteSecrets` row
with the error), and `ComputeCurrentArtifactHashesAsync` leaves those sites out of its map,
which the pending-updates evaluator reads as *Out of date*. Sites with no secret references
never touch the store and are unaffected. Before this, one `GetAsync` exception aborted the
whole sweep and the fleet-wide pending-updates check.
- *Removal-only deploys are audited.* The site's reply gained `SecretsRemoved` (proto
`ArtifactDeploymentResponseDto.secrets_removed = 7`, names only; absent/empty reads as null):
the `deployed_secrets` rows that deploy soft-removed. Central carries it on
`SiteArtifactResult.SecretsRemoved` and writes the `DeploySiteSecrets` row whenever it is
non-empty, with a `removed` list. An old site never sends it, so its removals stay unaudited
centrally, as before.
- *CLI exit code.* `site deploy-artifacts` and `deploy artifacts` exit `1` when the summary's
`failureCount` is non-zero (same rule `deploy site` already had).
- *Hub-era site-a secrets (review M8).* The retired hub had copied six `hub-gate-*`/`shared-gate-*`
test secrets (2026-08-07) into both rig site-a nodes' local stores. The mirror deliberately
never deletes a name it did not ledger, so they were removed once by hand on the rig (both
nodes stopped together; see the live-gate doc). QA and vd03 never ran the hub
(`Secrets:Replication:Enabled=false`), so nothing like this exists there.
- **Standby mirror failures reach central (follow-up, review finding M1).** A `DeployedSecretsMirror`
failure on the standby (e.g. a KEK mismatch) was only logged on that node — site health reports
are sent by the active node only. Each node now records its current failure set (names +
messages, never values) in a new, **12th LocalDb-replicated table `secret_mirror_status`** (one
row per node keyed by `node_name`), written only when the set changes plus once per start-up; a
per-node `SecretMirrorStatusReporter` puts every node's row on the active node's report as
`SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
**Stale rows (2026-09-29):** each node also re-writes its row every
`SiteRuntime:SecretMirrorStatusRefreshInterval` (10 min) as a heartbeat, and the reporter drops
and deletes rows not refreshed for `SiteRuntime:SecretMirrorStatusStaleAfter` (1 h), so a
decommissioned or renamed node no longer leaves its row on the Health page.
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
@@ -151,3 +151,46 @@ rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads.
Tests before the rig run: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors; Commons.Tests 875/875,
SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.
## 7. Follow-ups run (2026-09-28)
Code under test: branch `fix/site-secrets-followups` — per-site secret-store outage handling and
removal-only audit rows (`29049c4e`), artifact-deploy CLI exit codes (`ef03a0dc`), per-node mirror status on
the site health report (`0289d511`, merged in `4e640918`). Image rebuilt from the worktree with
`bash docker/build.sh`, all 8 nodes recreated from the primary checkout's compose file.
**Hub-era secrets removed from site-a (review M8), before the run.** The retired gRPC hub had left nine rows
dated 2026-08-07 in both site-a nodes' local stores (`/app/data/scadabridge-secrets.db`, table `secret`):
six live (`hub-gate-smoke-2`, `hub-gate-smoke-3`, `shared-gate-051`, `shared-gate-2`, `shared-gate-2b`,
`shared-gate-final`) and three tombstones (`expander-gate-pepper`, `hub-gate-smoke-1`, `shared-gate-1`).
Nothing in the rig config referenced them. With the site-a pair stopped together, each node's file was edited
from a throwaway `python:3.13-alpine` container mounting that node's data volume (`DELETE … WHERE name IN (…)
AND updated_utc < '2026-08-08'`, then `wal_checkpoint(TRUNCATE)`); 9 rows deleted per node, only the `sb-gate-*`
gate tombstones left. Both nodes came back ready, site-a-a active. site-b/site-c never ran the hub and had no
such rows; QA and vd03 never enabled replication.
**The gate grew to nine checks:**
- new check 5 (M1): site-a-b is recreated with the site-b/site-c dev KEK (a compose override, removed again in
the same check and in teardown), and central's `health site --identifier site-a` must show node-b failing
`sb-gate-mxgw` while node-a is clean, then both clean once site-a-b has site-a's KEK again;
- check 6 (was 5) also asserts the reply's `secretsRemoved` names the dropped secret;
- checks 7 and 8 (were 6 and 7) also assert the CLI exits 1;
- check 9 (was 8) now expects five `DeploySiteSecrets` rows — the removal-only deploy of check 6 writes one,
with `removed: [sb-gate-mxgw]`.
`bash docker/verify-site-secrets.sh` → **PASS 9/9**. Before the check, central reported both site-a node
rows with no failures; site-b and site-c (unreplicated) report their active node only. With the wrong KEK,
node-b's `sb-gate-mxgw` failure appeared on central's site-a report 57 s after site-a-b was back in the cluster
(mirror pass 15 s + status reporter 30 s + health report 30 s), and cleared 57 s after the KEK was restored.
Standby convergence: check 3 = 4.3 s, check 4 = 4.2 s, check 6 = 15.1 s. Audit: the 5 expected rows, 5 distinct
deployment ids; 0 plaintext/ciphertext hits in all 8 node logs and the audit payloads. Teardown left all three
sites *Current* and site-a-b on site-a's KEK.
Not exercised live: the central secret-store outage (review M5) — taking the shared SQL store down on the rig
also takes down central's configuration database. Covered by unit tests (`SiteSecretsCollectorTests`,
`ArtifactDeploymentServiceTests.StoreOutage_*`, `ComputeCurrentHashes_StoreOutage_*`,
`RetryForSite_StoreOutage_*`).
Tests: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors / 0 warnings; every non-Playwright test project green
(CLI.Tests 553 after updating `DeploymentDeadlineParityTests`' stub server to reply with a real zero-failure
summary — a bare `{}` now reads as an invalid artifact-deploy summary and exits 1). Playwright not run.
@@ -0,0 +1,55 @@
# System review remediation (2026-10-21)
Decision and execution record. Source: eight read-only parallel slice reviews of `main @ 513fa1aa` (findings in the review directory `00-SUMMARY.md` + `01..08`), excluding items already known from the 2026-09-09 arch review, the deferred-work register and CLAUDE.md. Remediation base for wave 2 docs: `1627513f`.
## Scope
8 slices (SiteRuntime; site data path DCL/S&F/event log; Communication + cluster; Deploy/Template/ScriptAnalysis/Transport; Management/InboundAPI/Security/ESG/Host; Audit/SiteCalls/Outbox/Notify/KPI/Health; Central UI + CLI; ConfigDB/Commons + gap inventory). ~107 findings: 4 High, ~30 Medium, ~73 Low. Recurring themes: non-collectible Roslyn loads, tracked whole-graph template reads for id/name lookups, uncached LDAP bind + N+1 role mapping, reconciliation paths that replay everything row-at-a-time, options validated but never read, sticky process-wide failure caches.
## Owner decisions
- **CLI auth = bearer per invocation.** One LDAP bind at `POST /auth/token`, then `Authorization: Bearer` on every management/audit/debug-stream call; Basic remains the fallback. Roles in a token are authoritative until expiry (<= `JwtExpiryMinutes`, 15); rotating `JwtSigningKey` invalidates all tokens; `StartupValidator` requires a >= 32-byte key on Central; `/auth/token` answers 404 under `DisableLogin`.
- **`Database:MachineDataDb` removed** (a required boot key nothing read).
- **Config audit stores summaries + optional retention** (script body = length + SHA-256; `AfterStateJson` capped 256 K; `ConfigAudit:RetentionDays` default 0 = keep).
- **Bound Test Run gate:** Test Run against a deployed instance needs Deployer + site scope and writes an audit row (closes the Flow Debugger F2/F3 asymmetry).
- **Overrides seed, not fire:** persisted static overrides loaded at actor start seed script state without firing scripts; alarms still evaluate.
## Lanes (one line each)
- **A SiteRuntime:** actor-name gate (`ActorNameRules`) + guarded `ActorOf` so a bad name can never restart the singleton (High); overrides republished/seeded on start; per-site serialised artifact apply with 10-min watchdog; metadata-only reconcile sweep; batched `Route.To().Get/SetAttributes`; DCL connection actor names ASCII-sanitised with collision rejection; startup warm/parallel compile.
- **B Scripts:** compile-cache eviction counter/warning + configurable bound (`ScriptCompileCacheMaxEntries`), faulted compiles never cached, scheduler slot reuse, shed-log rate limiting, native-alarm deletes batched into the flush.
- **C DCL:** same-adapter reconnect advances the session generation at issue time (High); MxGateway subscription ids `{session}:{handle}`; `IDataConnection.WriteBatchAndWaitAsync` removed; `MxAlarmStreamReopenDebounce` 200 ms; `ClientNamePrefix` now live; OPC UA discovery async and bounded by `OperationTimeoutMs`.
- **D Store-and-forward / event log:** per-lane ranked paging, gauge re-seed; event-log reads off the write lock with `KeywordSearchMaxWindow`; cap-trim count refresh every 10th batch, one vacuum per tick.
- **E Communication:** failback probe is a real readiness RPC (sentinel `$scadabridge-failback-probe`, 5-min probation); site-address refresh single-flight with deadline + sequence; streaming consumers re-resolve addresses on reopen; reconcile cancellation flows from the gRPC call; malformed address keeps the old channel (High); dead `SiteContacts` code deleted.
- **F Script analysis / Template Engine:** compile gate no longer emits/loads assemblies; trust verdicts cached by code hash; `BlockingWaitAnalyzer` warning; `ActorNameRules` at save; lean template projections.
- **G Deployment:** hash-only revision computation; single `FlattenSession` for bulk deploy pre-filter + prepare; reconcile fixed round trips with 3 retries; system artifact deployment records purged under `TerminalDeploymentRecordRetention`.
- **H Site audit:** reconciliation pull serves only un-acked rows; new site-only `PushAbandoned`/`Rejected` states; `IngestAck.rejected_event_ids` (field 2); persisted high-water; set-based paged central ingest (High); paged purge; fail-fast writer; per-site state pruning.
- **I Site calls / outbox / notify / KPI:** node-local `OperationTrackingCentralAck` ledger for un-acked pull; 128-row upsert chunks; conditional-aggregate KPI queries; outbox per-attempt timestamps, Pending/Retrying split claim, `DispatchSweepBudget` 2 min; SMTP `MaxConcurrentConnections` clamps Email parallelism; OAuth2 token cache eviction; KPI fold hour-at-a-time.
- **J Host / Inbound / ESG:** single-flight method compile and warm-up service, relay nonce single-use, in-memory request buffering, `ApiMethodCache` pruning, JSON GET/DELETE query params, central definition cache (`DefinitionCacheTtl` 10 s), secret-store fault vs missing PSK, NodeOptions-bound ports.
- **K Security / CLI:** bearer-per-invocation, `RoleMappingCache` (30 s, invalidated post-commit by the management handlers), 10 s display-state memo, dead refresh members removed from `JwtTokenService`.
- **L Central UI:** blocking-wait warning as Monaco marker, bounded script-analysis payload/concurrency, Topology behind `PollGate`, lean projections, bounded caches/rows, safe-invoke fire-and-forget, negative TTL for a faulted pending-updates flight.
- **M Management:** ListInstances SQL paging/prefix search, 409 `IMPORT_IN_PROGRESS` single-flight for large bodies, `CompileForPublish` once per save, site address + native-alarm-source name validation, terminal `PendingSecuredWrites` purge (`SecuredWriteTerminalRetention` 90 d).
- **N Transport:** import validates site gRPC addresses and actor-safe names as blockers; compressed decrypted content pinned on the session and zeroed on every release; template library read once; `StaleInstanceIds` covers derived/composing templates; `Transport:SchemaVersionMajor` removed.
- **O/P ConfigDB:** deterministic config-audit paging, `InstanceLocator` single projected join, dead repository members deleted, config-audit purge + `ConfigAuditOptions` binding, migration `ReviewRemediationIndexes`.
- NuGet: `System.Security.Cryptography.Xml`/`Pkcs` and SignalR.Common pinned to 10.0.13.
## Adversarial-review catches (fixed before merge)
- **C reconnect race:** the first fix advanced the session generation when the reconnect *result* was applied, so a new session dying before the actor processed that result had its `Disconnected` (stamped with the old generation) dropped as stale — leaving the actor Connected on a dead session with frozen tags. The generation now advances when the reconnect attempt is issued.
- **H central-fault refusals:** central answers DB faults with an empty/partial ack, which the new refusal counting treated as per-row refusals (a SQL maintenance window would mark rows `Rejected`). Central now names only row-intrinsic SQL failures in `IngestAck.rejected_event_ids`; the served high-water mark is persisted so owed rows retire after a site restart; transient faults never count toward reconciliation abandonment.
- **A data-connection actor names:** the actor-name gate covered instances/scripts/alarms but not DCL connection actors; names are sanitised and collisions rejected at deploy.
- **I outbox double-send:** the new Pending/Retrying split claim back-filled Pending with an offset (`Skip(n)`); an operator Retry flipping a Parked row to Pending mid-claim shifted the offset so one notification was claimed twice and delivered twice. The back-fill is now a keyset continuation plus a distinct-id guard.
- Also: failback probe sentinel changed to an illegal actor name; alarm aggregator reconcile no longer reopens during a pending backoff reconnect (double subscribe); `RoleMappingCache` invalidation after commit (lane K review M1).
## Deferred to the register
Rows 128-135 of `docs/plans/2026-07-08-deferred-work-register.md`: TLS production profile; `ExternalDefinitionLookupCache` invalidation; whole-graph load on template member add/update (04/P6 partial); idle `SiteStreamGrpcClientFactory` client eviction; compile-cache eviction Meter; dedicated rejected-rows health field; oversize audit row wedge (accepted); possibly redundant config-audit indexes. Row 36 annotated (same free-text `LIKE` shape on Instances/config audit; ListInstances now prefix-paged). Row 117 closed (already fixed by arch-review MG-6).
## Rollout notes
- **Not deployed:** the docker rig is rebuilt from main; QA (vq03/vq04) and prod (vd03) are not deployed.
- **Migration:** `ReviewRemediationIndexes` adds `AuditLog` `IX_AuditLog_Kind_Occurred` and `IX_AuditLog_Actor_Occurred` (partition-aligned) plus three composite `AuditLogEntries` indexes. Production applies `docs/plans/sql/ReviewRemediationIndexes.sql` by hand (index builds on a large partitioned `AuditLog`: schedule off-hours). Not yet applied to QA/prod.
- **Mixed version:** upgrade central first (new `IngestAck` field 2 and proto additions are additive; an old central ignores `rejected_event_ids` and an old site never rejects). Pair upgrades stay both-nodes-together for LocalDb sites.
- **Config changes:** CLI hosts need a central that serves `/auth/token` (older centrals fall back to Basic automatically); Central must have a >= 32-byte `JwtSigningKey` or fails startup validation; remove `Database:MachineDataDb` and `Transport:SchemaVersionMajor` from appsettings (ignored if left).
- **New keys (all defaulted):** `ScadaBridge:ConfigAudit:{RetentionDays 0, PurgeBatchSize 5000, MaxBatchesPerTick 200, PurgeInterval 1d}`; `Security:RoleMappingCacheSeconds 30`; `SiteEventLog:{KeywordSearchMaxWindow 7d, CapTrimCountRefreshBatches 10}`; `AuditLog` site writer/reconciler `{MaxRejectedPushAttempts 10, PurgeBatchSize 5000, PurgeVacuumPagesPerBatch 1000, MaxPagesPerTick 50}`; `SiteRuntime:ScriptCompileCacheMaxEntries 1024`; `NotificationOutbox:DispatchSweepBudget 2m`; `ManagementService:{SecuredWriteTerminalRetention 90d, SecuredWriteTerminalPurgeInterval 1h}`; ESG `DefinitionCacheTtl 10s` (central); DCL `MxAlarmStreamReopenDebounce 200ms`; `DeploymentManager:TerminalDeploymentRecordRetention` governs system artifact deployment record purge.
- **Behaviour changes operators will notice:** `instance list --search` is prefix-only; Test Run against a deployed instance needs Deployer + site scope; deploys reject non-actor-safe names (rename before redeploy); a redeployed instance's persisted overrides no longer re-fire value-change scripts on start.
@@ -0,0 +1,71 @@
-- ReviewRemediationIndexes (2026-10-22). Run with QUOTED_IDENTIFIER ON (sqlcmd -I, or keep
-- this header): AuditLog has persisted computed columns / filtered indexes and SQL Server
-- raises error 1934 otherwise.
-- AuditLog is monthly-partitioned; the two AuditLog indexes are created ON
-- ps_AuditLog_Month(OccurredAtUtc) so they stay partition-aligned and the partition-switch
-- purge keeps working. Building them scans the whole AuditLog: on a large production table
-- run in a quiet window, and on Enterprise edition add WITH (ONLINE = ON) to the two
-- AuditLog CREATE INDEX statements (and to the AuditLogEntries ones if that table is large).
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
CREATE INDEX [IX_AuditLogEntries_EntityType_Timestamp] ON [AuditLogEntries] ([EntityType], [Timestamp] DESC);
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
CREATE INDEX [IX_AuditLogEntries_Timestamp_Id] ON [AuditLogEntries] ([Timestamp] DESC, [Id] DESC);
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
CREATE INDEX [IX_AuditLogEntries_User_Timestamp] ON [AuditLogEntries] ([User], [Timestamp] DESC);
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred
ON dbo.AuditLog (Kind, OccurredAtUtc DESC)
ON ps_AuditLog_Month(OccurredAtUtc);
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred
ON dbo.AuditLog (Actor, OccurredAtUtc DESC)
ON ps_AuditLog_Month(OccurredAtUtc);
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20261022133907_ReviewRemediationIndexes'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20261022133907_ReviewRemediationIndexes', N'10.0.7');
END;
COMMIT;
GO
+145 -30
View File
@@ -255,13 +255,21 @@ has no anchor and falls back to the unbounded scan (correctness over speed).
A SQLite database file on each site node, alongside the Store-and-Forward
buffer. Same schema as central minus `IngestedAtUtc` (irrelevant at the source),
plus a `ForwardState` column with values `Pending | Forwarded | Reconciled` that
drives the telemetry loop and reconciliation pull. `SourceNode` is stamped by the
drives the telemetry loop and reconciliation pull — and, since review 2026-10-21
(06/PERF-1, 06/IMPL-1), two site-only **push-exit** states, `PushAbandoned` and
`Rejected`, for rows that left the telemetry push queue *without* a central ack.
`Forwarded` means "central acked it"; `Pending`, `PushAbandoned` and `Rejected`
together are **owed** to central: the reconciliation pull serves exactly those,
a covering cursor retires them to `Reconciled`, and the retention purge never
drops them. The extra states are strings in the site sidecar only (no schema
change, never on the wire); a downgraded node ignores rows in a state it does not
know. `SourceNode` is stamped by the
writing node itself (`node-a` / `node-b`) at append time and travels with the row
through telemetry and reconciliation unchanged.
**Site SQLite retention rule (hard invariant):**
> A row is eligible for purge only when both `OccurredAtUtc < retention threshold` AND `ForwardState IN ('Forwarded', 'Reconciled')`. Pending rows are never purged.
> A row is eligible for purge only when both `OccurredAtUtc < retention threshold` AND `ForwardState IN ('Forwarded', 'Reconciled')`. Pending rows — and the owed push-exit states `PushAbandoned` / `Rejected` — are never purged.
A prolonged central outage will grow the site audit table indefinitely until
central is reachable again. This is intentional — losing audit rows to make
@@ -302,8 +310,36 @@ round-trip on the hot path.
A `SiteAuditTelemetryActor` singleton drives the forwarding loop: select up to
N `Pending` rows ordered by `OccurredAtUtc`, batch-send them to central via the
existing `SiteStream` gRPC channel as `IngestAuditEvents(events)`, and on
central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. Rejected IDs
stay `Pending` for the next sweep. Cadence is short (default 5 s) when
central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. IDs central
answered for but did not accept stay `Pending` for the next sweep. Central
additionally lists, in the ack's additive `rejected_event_ids` (sitestream.proto
`IngestAck` field 2), the ids it refused for a **permanent, row-intrinsic** SQL
reason (string truncation 8152/2628, NOT NULL 515, conversion/overflow
245/241/242/8114/8115/220/232, constraint 547). Only those are refusals: the site
bumps `AttemptCount` / `LastAttemptUtc` for them and, after
`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10), moves the row out of
the push queue as `Rejected`, counted on its own `scadabridge.site.audit.rejected_rows`
counter (not `SiteAuditWriteFailures` — the site's write succeeded). A row merely
absent from the accepted list — a SQL maintenance window, a timeout, an exhausted
ingest budget — is never counted, and an older central that does not send the field
never triggers it (review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
row central can never insert (e.g. an over-width column) sat at the head of every
batch forever, failing each set-based insert at central into the per-row fallback.
`Rejected` is not a drop: the reconciliation pull still serves the row, so a
transient refusal still lands, and a genuinely poison row is recorded centrally as
a `ReconciliationAbandoned` row before its cursor retires it. A transport fault (no
answer) is never counted as a refusal. Symmetrically, the reconciliation puller counts
only **row-intrinsic** failures (the same narrow set the push path reports as rejected:
truncation, NULL into NOT NULL, conversion/overflow, constraint) toward its
permanent-abandon threshold: a timeout, connection loss, deadlock, throttling error, or
schema drift after a missed central migration (207/208) holds the cursor and retries
without ever abandoning the row. And even a row-intrinsic failure abandons a row only on a
tick where at least one *sibling* row of the same page inserted — a page where every row
fails identically is a database problem, and abandoning it would let the site mark every
row `Reconciled` and purge it (review 2026-10-21 follow-up M1). A cached-lifecycle row whose tracking
snapshot is gone past `CachedTrackingGraceSeconds` leaves the cached drain as
`PushAbandoned` (it used to be marked `Forwarded`, which only worked because the
pull re-served every `Forwarded` row). Cadence is short (default 5 s) when
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
dispatcher.
@@ -317,7 +353,11 @@ collapse first-write-wins before the statement is built, duplicates *across*
packets are eliminated by the anti-semi-join, and any failure falls back to the
per-row / per-entry path — so the documented invariant that one bad row cannot
sink the rest of the batch still holds, it is simply no longer paid for on the
healthy path.
healthy path. A site `LastError` longer than the central `SiteCalls.LastError` column (1024
characters) is truncated to the column width on both the dual-write and the
reconciliation pull (`SiteCallColumnBounds`) — an over-width message used to fail the
whole dual-write with SQL 8152, so the audit row came back Rejected and the
`SiteCalls` row was never written by either path (review 2026-10-21 follow-up).
The set-based path binds its string parameters at the VALUE's own length, never
at the column width. Declaring the width makes the client TRUNCATE an over-long
@@ -349,10 +389,50 @@ still replied while the outer Asks are still waiting.
### Reconciliation pull (self-healing for missed telemetry)
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
asks each site for its oldest `Pending` row and pending count; if backlog is
non-draining (e.g., telemetry actor wedged), central issues a
`PullAuditEvents(sinceUtc, batchSize[, afterId])` and inserts-if-not-exists.
Same self-healing pattern as Site Call Audit's reconciliation of `SiteCalls`.
issues `PullAuditEvents(sinceUtc, batchSize[, afterId])` against each site and
inserts-if-not-exists. Same self-healing pattern as Site Call Audit's
reconciliation of `SiteCalls`.
**The pull serves only rows central has not acknowledged (review 2026-10-21
06/PERF-1).** `ReadPendingSinceAsync` returns `Pending`, `PushAbandoned` and
`Rejected` rows — never `Forwarded`. It used to serve `Forwarded` rows too, so
every row the push had already delivered was re-shipped and re-inserted
(row-at-a-time) one tick later. That capped the pull at 256 rows per 5-minute tick
(≈0.85 rows/s per site, 60× below the push), so any site above that rate fell
permanently behind: a genuinely lost row waited behind days of already-forwarded
rows until the 7-day site purge, the stalled badge latched true on a healthy busy
site, and every central failover (cursor reset to `MinValue`) restarted a
multi-day replay (06/COR-3, resolved by this change — the re-pull after a reset is
now only the owed set). Central ingests each page with ONE set-based
`InsertManyIfNotExistsAsync` (per-row fallback only when the batch throws, so the
poison-row retry/abandon semantics are unchanged) and **pages within a tick** — up
to `AuditLog:Reconciliation:MaxPagesPerTick` (default 50) pulls while the site
reports `MoreAvailable` and the cursor advances, mirroring `SiteCallAuditActor`.
**Mixed versions:** a pre-PERF-1 site still serves `Forwarded` rows; central
tolerates that unchanged (idempotent on `EventId`, merely more pages).
**Rollback caveat.** Rolling a site back to a pre-PERF-1 binary strands any rows already in
`PushAbandoned` / `Rejected` on disk: the old code neither pushes, serves, retires nor purges
a state it does not know, so they sit in the audit file (harmless, but never delivered and
never purged) until the node is upgraded again — at which point the pull serves them
normally. Nothing else in the file changes shape (the `audit_site_state` table is ignored by
an old binary).
**Oversize rows (accepted, documented).** A single site audit row larger than the 32 MB gRPC
ceiling (`GrpcMessageLimits.MaxMessageBytes`) cannot be pushed or pulled and would wedge the
head of both paths. It is unreachable under the payload caps (8 KB / 64 KB on error by
default) unless `DefaultCapBytes`/`ErrorCapBytes` are configured into the tens of megabytes;
rows above the 2 MB batch budget but under the ceiling already travel alone.
**Per-site state follows the site list (review 2026-10-21 06/MEM-2).** Each tick drops
the cursor, non-draining counter and stalled latch of any site no longer returned by
`ISiteEnumerator` (a latched site publishes `Stalled=false` as it goes; an EMPTY enumeration
prunes nothing, since it is as likely a transient lookup glitch), and the
`PullAuditEvents` / `PullSiteCalls` / `PullTraceEvents` invokers evict and dispose any
cached `(site, endpoint)` gRPC channel that is not a live site's primary or fallback
address on every enumeration. The in-memory abandoned-`EventId` short-circuit set is
capped at 10 000 entries (cleared when full — it is an optimisation only; the
deterministic synthetic id is what keeps re-abandonment idempotent).
**`PullTraceEvents` is a point read, not a reconciliation path (2026-09-05).** The
Flow Debugger's value-change run cannot get its steps inline (the device echo
@@ -407,10 +487,12 @@ the legacy inclusive `>=` read as the compatible fallback. The cursor stays in-m
and resets on singleton restart; idempotent `InsertIfNotExistsAsync` still absorbs any
re-pulled duplicates, so exactness is an optimization, not a correctness dependency.
Unlike Site Call Audit, this actor issues ONE pull per tick rather than paging within
it — a lagging drain is meant to surface as the stalled signal. The id tiebreak is
what makes that safe against a same-instant burst larger than one batch: the cursor
advances on every tick even when the timestamp cannot.
Like Site Call Audit, the actor pages within a tick (bounded by `MaxPagesPerTick`), so
the stalled signal fires only when a site still reports `MoreAvailable` after the page
cap — or its cursor is held back by a row that keeps failing — on
`StalledAfterNonDrainingCycles` consecutive ticks: a genuine non-drain, not a busy
site. The id tiebreak keeps paging safe against a same-instant burst larger than one
batch: the cursor advances on every page even when the timestamp cannot.
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
@@ -421,21 +503,35 @@ and, being `Reconciled`, purged on age. That is silent audit loss, and it is a f
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
carries a second, insertion-order bound:
> A `Pending` row retires only if its insertion order is at or below the high-water
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
> push path, which is proof independent of the pull.
> An owed row (`Pending` / `PushAbandoned` / `Rejected`) retires only if its insertion
> order is at or below the high-water mark of rows this site node has actually served
> from `ReadPendingSinceAsync` (SQLite `rowid`). `Forwarded` rows are not touched by the
> cursor at all — central ACKED them over the telemetry push path, they are already
> purge-eligible, and the pull never served them.
The bound is per-process, so after a site-node restart the first pull retires only
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
The bound is persisted in a node-local `audit_site_state` key/value table in the audit
file (loaded at start-up, clamped to the live `MAX(rowid)`), so a site-node restart no longer
resets it — before that, a healthy site whose pull serves nothing could leave already-served
`PushAbandoned`/`Rejected` rows owed (and counted on the backlog) indefinitely. It stays
conservative in the safe direction (retirement can be delayed, never a row lost). The gRPC handler's ordering
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
about to serve.
about to serve. The mark is advanced under the writer's write lock — the same lock
the purge's rowid clamp runs under — so a clamp can never be overwritten with a stale-high
value by a concurrent pull (review 2026-10-21 follow-up L1).
**The bound protects a late-stamped row for one pull cycle, not forever.** A row inserted
after a batch was served but stamped *below* central's cursor has a rowid above the mark,
so the next cursor cannot retire it — but the pull never serves it either (the keyset read
is already past its timestamp), and as soon as a later pull serves any newer row the mark
rises above it and the following cursor retires it. Its delivery therefore relies on the
telemetry **push**, which drains every `Pending` row oldest-first regardless of the pull
cursor and will normally have acked it (`Forwarded`, untouched by the cursor) within that
cycle; the served-rowid bound only buys the push that window.
**Liveness note (accepted, documented).** A row that was served but never covered by a
later cursor — central reconciliation stopped for good, or the bound reset over a
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
later cursor — central reconciliation stopped for good — stays owed (`Pending` /
`PushAbandoned` / `Rejected`) indefinitely, and the site retention purge never purges
an owed row. That is the hard `ForwardState` invariant
working as intended (an unacknowledged row is not droppable), but it means the site
store's floor depends on reconciliation actually running, not merely on the retention
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
@@ -574,11 +670,23 @@ MS SQL for direct-write events). Unredacted secrets never persist.
(disk full, schema lock, transient IO), events buffer in a small in-memory
ring (default 1024 rows); oldest are discarded with a Site Event Log warning
per drop.
- **A saturated writer fails fast; it never parks callers.** The site writer's
bounded queue (`AuditLog:SiteWriter:ChannelCapacity`, default 4096) used to
make `WriteAsync` *await* room once full — and because every hot-path caller is
fire-and-continue, a writer stuck behind a long purge accumulated an unbounded
number of parked waiters, each holding its event, while reporting healthy. A
full queue now faults the write immediately with
`SqliteAuditWriterSaturatedException`; `FallbackAuditWriter` treats it like
any other primary failure — the event goes to the drop-oldest ring above and
`SiteAuditWriteFailures` is bumped — so the degradation is the documented
best-effort drop with a counter, never unbounded memory (review 2026-10-21
06/MEM-1). Audit failure still never aborts the action.
- **Reconciliation as fallback.** If two consecutive reconciliation cycles
report a non-draining backlog, the supervisor restarts the telemetry actor
and a `SiteAuditTelemetryStalled` event fires.
still report a non-draining backlog after paging up to `MaxPagesPerTick`, a
`SiteAuditTelemetryStalled` event fires.
- **Permanent-abandonment record.** A reconciliation pull row that fails to
insert on every retry up to the permanent-abandon threshold is dropped and the
insert with a row-intrinsic error on every retry up to the permanent-abandon
threshold, while a sibling row of its page inserts, is dropped and the
cursor advances past it (so one broken row can't block the site forever). The
loss is not silent: alongside the Critical log line the actor writes ONE
synthetic `ReconciliationAbandoned` audit row (fresh `EventId`, `Status=Failed`,
@@ -655,9 +763,16 @@ MS SQL for direct-write events). Unredacted secrets never persist.
node still purges soon after start), then every
`AuditLog:SiteRetention:PurgeInterval` (default 24 h, clamped to ≥ 1 min).
Each tick calls `ISiteAuditQueue.PurgeExpiredAsync(UtcNow − RetentionDays)`,
which deletes eligible rows in one transaction (sidecar first, then the
canonical row) and then runs `PRAGMA incremental_vacuum` to return the freed
pages to the OS. `AuditLog:SiteRetention:RetentionDays` defaults to 7 and is
which deletes eligible rows **in pages** of at most
`AuditLog:SiteWriter:PurgeBatchSize` rows (default 5000) — one transaction per
page, sidecar first, then the canonical row — and after each page runs a
**bounded** `PRAGMA incremental_vacuum(N)` (`AuditLog:SiteWriter:PurgeVacuumPagesPerBatch`,
default 1000 pages) to return freed pages to the OS. The write lock is taken per
page and released between pages, so the batched hot-path writer interleaves with a
multi-million-row purge instead of stalling behind one giant DELETE plus an
unbounded vacuum (review 2026-10-21 06/MEM-1 — the same hazard register row 34
closed for `site_events`). One call still removes every eligible row; only the
lock grain changed. `AuditLog:SiteRetention:RetentionDays` defaults to 7 and is
clamped to `[1, 90]`. Respects the hard `ForwardState` invariant — a `Pending`
row is never purged on age alone; only `Forwarded`/`Reconciled` rows older than
the cutoff are removed. Per-tick failures are swallowed and logged so a transient
+16 -13
View File
@@ -2,7 +2,7 @@
## Purpose
The CLI is a standalone command-line tool for scripting and automating administrative operations against the ScadaBridge central cluster. It connects to the Central Host's HTTP Management API (`POST /management`), which dispatches commands to the ManagementActor. Authentication and role resolution are handled server-side — the CLI sends credentials via HTTP Basic Auth. The CLI provides the same administrative capabilities as the Central UI, enabling automation, batch operations, and integration with CI/CD pipelines.
The CLI is a standalone command-line tool for scripting and automating administrative operations against the ScadaBridge central cluster. It connects to the Central Host's HTTP Management API (`POST /management`), which dispatches commands to the ManagementActor. Authentication and role resolution are handled server-side — the CLI exchanges its credentials once per process for a short-lived bearer token (`POST /auth/token`), falling back to HTTP Basic Auth on servers without bearer support. The CLI provides the same administrative capabilities as the Central UI, enabling automation, batch operations, and integration with CI/CD pipelines.
## Location
@@ -13,31 +13,34 @@ Standalone executable, not part of the Host binary. Deployed on any machine with
## Responsibilities
- Parse command-line arguments and dispatch to the appropriate management operation.
- Send HTTP requests to the Central Host's Management API endpoint with Basic Auth credentials.
- Send HTTP requests to the Central Host's Management API endpoint with a bearer token (or Basic Auth credentials on older servers).
- Display structured responses from the Management API.
- Support both JSON and human-readable table output formats.
## Technology
- **Argument parsing**: `System.CommandLine` library for command/subcommand/option parsing with built-in help generation.
- **Transport**: HTTP client connecting to the Central Host's `POST /management` endpoint. Authentication is via HTTP Basic Auth — the server performs LDAP bind and role resolution.
- **Transport**: HTTP client connecting to the Central Host's `POST /management` endpoint. Authentication is via a bearer JWT obtained from `POST /auth/token` (or HTTP Basic Auth) — the server performs the LDAP bind and role resolution.
- **Serialization**: Commands serialized as JSON with a type discriminator (`command` field). Message contracts from Commons define the command types.
## Authentication
The CLI sends user credentials to the Management API via HTTP Basic Auth:
The CLI authenticates with a bearer token exchanged once per process (review 2026-10-21 05/U1, 05/P1):
1. The user provides credentials via `--username` / `--password` options.
2. On each request, the CLI encodes credentials as a Basic Auth header and sends them with the command.
3. The server performs LDAP authentication, group lookup, and role resolution — the CLI does not communicate with LDAP directly.
4. Credentials are not stored or cached between invocations. Each CLI invocation requires fresh credentials.
1. The user provides credentials via `--username` / `--password` (or `SCADABRIDGE_USERNAME` / `SCADABRIDGE_PASSWORD`).
2. On the first management call of the process, the CLI posts the credentials to `POST /auth/token`. The server performs the LDAP bind, group lookup and role resolution, and returns a short-lived HS256 JWT carrying the username, roles and permitted sites (lifetime `JwtExpiryMinutes`, default 15 min). That is the invocation's **only** LDAP bind.
3. Every later call in the process — `POST /management`, the audit REST endpoints, and the `debug stream` hub (including its automatic reconnects) — sends `Authorization: Bearer <token>`; the server validates it locally with no LDAP bind. A token near expiry is re-exchanged.
4. A bearer call answered `401` re-exchanges once (unless the token was minted for that very call) and, if still refused, retries with HTTP Basic and stays on Basic for the rest of the process.
5. **Basic fallback.** A server with no `/auth/token` (404/405), a login-disabled deployment (404), or an older server whose token response lacks `management_bearer: true` (its tokens are accepted nowhere) is remembered as Basic-only for the process: each call carries a Basic header and the server binds LDAP per request. A transient exchange error (5xx) uses Basic for that call only. Bad credentials (`401`) or a throttle lockout (`429`) from the exchange are reported as-is and are **not** retried with Basic (that would only add a second failed bind against the login throttle).
6. The token lives **in memory only** — never written to disk, logged or echoed — and dies with the process; each CLI invocation needs credentials. `SCADABRIDGE_AUTH_MODE=basic` disables the exchange (Basic on every call); `SCADABRIDGE_VERBOSE=1` prints a stderr note when a freshly issued token is refused and the CLI falls back to Basic. The client never follows HTTP redirects, so a 307/308 cannot replay the credential form elsewhere.
7. The CLI never talks to LDAP directly.
## Connection
The CLI connects to the Central Host via HTTP:
- **Management URL**: The URL of a central node's web server (e.g., `http://localhost:9001`). The management API is served at `POST /management` on the same host as the Central UI.
- **Failover**: For HA, use a load balancer URL in front of both central nodes. The management API is stateless (Basic Auth per request), so any central node can handle any request without sticky sessions.
- **Failover**: For HA, use a load balancer URL in front of both central nodes. The management API is stateless (a self-contained bearer token, or Basic Auth per request), and both central nodes share the JWT signing key, so any central node can handle any request without sticky sessions.
- **No Akka.NET dependency**: The CLI is a pure HTTP client with no Akka.NET runtime.
## Command Structure
@@ -356,7 +359,7 @@ scadabridge debug stream --id <id>
The `debug snapshot` command retrieves a point-in-time snapshot via the HTTP Management API.
The `debug stream` command streams live attribute values and alarm state changes in real-time using a SignalR WebSocket connection. The CLI connects to the `/hubs/debug-stream` SignalR hub on the central server, authenticates with Basic Auth, and subscribes to the specified instance. Events are printed as they arrive — JSON format (default) outputs one NDJSON object per event; table format shows streaming rows. Press Ctrl+C to disconnect.
The `debug stream` command streams live attribute values and alarm state changes in real-time using a SignalR WebSocket connection. The CLI connects to the `/hubs/debug-stream` SignalR hub on the central server, authenticates with the process's bearer token (the SignalR client sends it in the `Authorization` header of the negotiate request and WebSocket upgrade, and re-fetches it on every reconnect) — or Basic Auth on a server without bearer support — and subscribes to the specified instance. Events are printed as they arrive — JSON format (default) outputs one NDJSON object per event; table format shows streaming rows. Press Ctrl+C to disconnect.
Key behaviors:
- **Automatic reconnection**: Uses SignalR's `.WithAutomaticReconnect()` to re-establish the connection on loss.
@@ -505,7 +508,7 @@ Configuration is resolved in the following priority order (highest wins):
- **Connection failure**: If the CLI cannot connect to the management URL (e.g., DNS failure, connection refused), it exits with code 1 and a descriptive error message.
- **Command timeout**: Ordinary CLI commands use a 30-second client deadline. All instance/artifact/site deployment aliases defer the overall deadline to the server, retaining a bounded socket-connect timeout. Server Ask expiry returns HTTP 504; a client deadline is reported locally. Bundle and Flow Debugger commands retain their explicit timeout policies.
- **Authentication failure**: If the server returns HTTP 401 (LDAP bind failed), the CLI exits with code 1.
- **Authentication failure**: If the server returns HTTP 401 (LDAP bind failed at `/auth/token` or on a Basic call, or a bearer token that a fresh exchange could not replace), the CLI exits with code 1.
- **Authorization failure**: If the server returns HTTP 403, the CLI exits with code 2.
## Dependencies
@@ -513,13 +516,13 @@ Configuration is resolved in the following priority order (highest wins):
- **Commons**: Message contracts (`Messages/Management/`) for command type definitions and registry.
- **System.CommandLine**: Command-line argument parsing.
- **Microsoft.AspNetCore.SignalR.Client**: SignalR client for the `debug stream` command's WebSocket connection.
- **Management Service (#18)**: The CLI hits the central cluster via the existing HTTP Management API (`POST /management`), which dispatches to the ManagementActor. The `scadabridge audit` command group rides a parallel REST surface on the same Host (`GET /api/audit/query` and `GET /api/audit/export`), sharing HTTP Basic Auth with `/management` but bypassing the actor for read-only, keyset-paged / streaming workloads.
- **Management Service (#18)**: The CLI hits the central cluster via the existing HTTP Management API (`POST /management`), which dispatches to the ManagementActor. The `scadabridge audit` command group rides a parallel REST surface on the same Host (`GET /api/audit/query` and `GET /api/audit/export`), sharing authentication (bearer token or HTTP Basic) with `/management` but bypassing the actor for read-only, keyset-paged / streaming workloads.
- **Audit Log (#23)**: The `scadabridge audit query`, `audit export`, `audit tree`, and `audit backfill-source-node` subcommands target the centralized Audit Log component's REST endpoints (`GET /api/audit/query`, `GET /api/audit/export`, `GET /api/audit/tree`, `POST /api/audit/backfill-source-node`) on the Host's Management API surface; `audit verify-chain` is a client-side no-op today (hash-chain deferred to v1.x). Permission checks (`OperationalAudit`, `AuditExport`, `Admin`) are enforced server-side by `AuditEndpoints`.
## Interactions
- **Management Service (via HTTP)**: The primary runtime dependency. All operations except `debug stream` are sent as HTTP POST requests to the Management API endpoint on a central node, which dispatches to the ManagementActor.
- **Central Host**: Serves the Management API at `POST /management` and the debug stream SignalR hub at `/hubs/debug-stream`. Handles LDAP authentication, role resolution, and ManagementActor dispatch.
- **Central Host**: Serves the Management API at `POST /management` and the debug stream SignalR hub at `/hubs/debug-stream`. Issues bearer tokens at `POST /auth/token`; handles LDAP authentication, role resolution, and ManagementActor dispatch.
- **Debug Stream Hub (via SignalR WebSocket)**: The `debug stream` command connects to the `/hubs/debug-stream` hub on the central server for real-time event streaming. This is the only CLI command that uses a persistent connection rather than request/response.
+12 -2
View File
@@ -19,7 +19,7 @@ Central cluster only. Sites have no user interface.
- **The decision.** *Approve* is `Caution` (amber): approving executes a real write to live plant equipment, so it is not a benign primary action, but it is not destroying anything either. *Reject* **keeps** `Destructive` (red): it is terminal and irreversible for the request — the Operator must resubmit — and holding it red is what keeps the two opposite decisions visually distinct. Dropping *Reject* to a neutral button was considered and rejected: it would simply invert the bias, making declining look like the safe default.
- Colour is never the only signal — the button always carries its verb (see the label contract above), so the mapping is an accessibility affordance, not the meaning.
- **`DialogSeverity` is an additive overload**, not a replacement: the boolean form remains the signature for the ~34 existing call sites, which are genuinely two-state. Adding a fifth optional parameter to the boolean form instead would have silently un-matched every existing four-argument NSubstitute stub — the row-28 gotcha, where a stub that stops matching returns a null `Task` and the caller NREs with no compile error. Reach for the overload only when `Caution` is what you mean.
- **Pending updates rail indicator (2026-09-14, `PendingUpdatesIndicator`, design `docs/plans/2026-09-14-pending-updates-design.md`).** A block in the rail footer, placed above the user name (`MainLayout`'s `RailFooter`, `.rail-user`), summarising the fleet-wide `PendingUpdatesSnapshot` from Component-DeploymentManager.md's *Pending-updates aggregate*. Rendered only for a user holding **Deployer or Administrator** — the two roles that can act on it — and hidden for everyone else. States: *Checking for updates…* (first load), *Up to date* (muted, check icon), **Pending updates** with a warning count pill and a breakdown line (`N instances`, linking to `/deployment/topology` for a Deployer; `M sites`, linking to `/admin/sites` for an Administrator, plain text otherwise), or *Updates unavailable* (error styling, tooltip carries the failure message) on a faulted evaluation. A tooltip on the block reads *as of HH:mm UTC*. It polls `IPendingUpdatesCache.GetAsync()` every 30 s through `PollGate` — cheap, since the cache's own 60 s TTL means the fleet is actually re-flattened at most once per TTL fleet-wide, no matter how many circuits (or how often each one) poll it — disposes its timer with the circuit, and renders through `InvokeAsync`. A **site-scoped** user sees counts narrowed to their permitted sites only (`PendingUpdatesSnapshot.ForSites`, the same scope `SiteScopeService` applies to Topology), so the indicator and the Sites page never disagree about which sites a given operator can see.
- **Pending updates rail indicator (2026-09-14, `PendingUpdatesIndicator`, design `docs/plans/2026-09-14-pending-updates-design.md`).** A block in the rail footer, placed above the user name (`MainLayout`'s `RailFooter`, `.rail-user`), summarising the fleet-wide `PendingUpdatesSnapshot` from Component-DeploymentManager.md's *Pending-updates aggregate*. Rendered only for a user holding **Deployer or Administrator** — the two roles that can act on it — and hidden for everyone else. States: *Checking for updates…* (first load), *Up to date* (muted, check icon), **Pending updates** with a warning count pill and a breakdown line (`N instances`, linking to `/deployment/topology` for a Deployer; `M sites`, linking to `/admin/sites` for an Administrator, plain text otherwise), or *Updates unavailable* (error styling, tooltip carries the failure message) on a faulted evaluation. A tooltip on the block reads *as of HH:mm UTC*. It polls `IPendingUpdatesCache.GetAsync()` every 30 s through `PollGate` — cheap, since the cache's own 60 s TTL means the fleet is actually re-flattened at most once per TTL fleet-wide, no matter how many circuits (or how often each one) poll it — disposes its timer with the circuit, and renders through `InvokeAsync`. **A finished deployment refreshes it at once (2026-10-22):** the cache subscribes to `IDeploymentStatusNotifier`, drops its memoized snapshot on every `Success`/`Failed` status (a flight already running when that happens still answers its callers but is not memoized — `SingleFlightMemo.Invalidate`), and raises `IPendingUpdatesCache.Invalidated` once completions have been quiet for 1 s, so a bulk deploy's burst notifies once; the indicator re-reads on that signal (a reload the `PollGate` drops mid-poll is re-run once). Before this, a Deploy all left the pre-deploy count on the rail for up to the TTL plus one poll (~90 s). Only deployments run on the same central node are seen — one run through the peer still clears within the TTL. A **site-scoped** user sees counts narrowed to their permitted sites only (`PendingUpdatesSnapshot.ForSites`, the same scope `SiteScopeService` applies to Topology), so the indicator and the Sites page never disagree about which sites a given operator can see.
- **Rail nav scroll pane (2026-09-14).** `NavMenu.razor` wraps its items in a `<div class="rail-scroll">`; `site.css` (≥ lg only) fills the sticky `100vh` `#theme-rail` (`#theme-rail > .side-rail { height: 100% }`) and makes `.rail-scroll` the flexible, independently-scrolling region (`flex: 1 1 auto; min-height: 0; overflow-y: auto`), so a long expanded nav scrolls under the brand bar instead of pushing the rail footer (and now the pending-updates indicator) off-screen. Below lg the rail is inline and scrolls with the page, unchanged. The `ZB.MOM.WW.Theme` kit itself is untouched — this is app-local CSS/markup layered on top of it (deferred-work register row 123 proposes upstreaming it for the kit's other consumers).
## Review hardening (2026-09-09)
@@ -28,10 +28,19 @@ Central cluster only. Sites have no user interface.
- Manual failover goes through a scoped `AuthorizedManualFailoverService` that refreshes the circuit's Administrator grant before calling the host singleton. The component's visibility policy is an additional presentation guard.
- Flow Debugger retry/discard validates the current Deployer grant and resolves site scope inside its service. Scoped callers cannot relay an unknown site. Refresh merges preserve Replay's event ordering and append only rows known exclusively to the live run.
- Health and Alarm Summary observe the whole posted refresh task and release their poll gate even when renderer dispatch fails during disposal.
- The Topology initially loads permitted sites. Expand a site and select **Load instances** to load that site's area metadata and first root-instance page; each expanded area has its own loader. **Previous** and **Next** replace that branch's 50-instance page. Queries are untracked and omit member collections; templates are not loaded. Live refresh and staleness checks cover loaded pages. Search and counts explicitly describe loaded data.
- The Topology initially loads permitted sites **and their area metadata** (Gitea #39 — every permitted site's areas are read at open, out-of-scope sites never) and opens with each site expanded to its top-level areas (`TreeView.InitiallyExpanded`; a browser's saved expansion state always wins). Instances stay lazy: select **Load instances** to load the first root-instance page of a site; each expanded area has its own loader. **Previous** and **Next** replace that branch's 50-instance page. Queries are untracked and omit member collections; templates are not loaded. Live refresh and staleness checks cover loaded pages. Search and counts explicitly describe loaded data. **Deploy all…** (Gitea #40): the Site and Area node context menus offer a subtree bulk deploy (Deployer role + current site scope re-checked at click). The page asks `DeploymentService.PreviewSubtreeDeployAsync` for the instance count and Disabled count (a cheap read, no staleness probe); a zero-instance subtree shows an info toast and no dialog; otherwise `DeployAllDialog` (via `IDialogService.ShowAsync`) shows the counts and a **Force redeploy up-to-date instances** checkbox with confirm label **Deploy all**. Confirm calls `DeploySiteAsync` (site node) or `DeployAreaAsync` (area node — the area plus all descendant areas) with the Force choice — same rules as the CLI/management bulk deploy (Disabled always skipped, up-to-date skipped unless forced, not all-or-nothing) — then toasts `N deployed, M skipped, K failed` (an error toast naming the first three failures when any failed) and reloads the tree. The menu item is disabled while another page action is in flight.
- `TemplateFolderTree` compares immutable value snapshots of folder/template hierarchy and extra children before rebuilding. In-place renames, moves, derived-state changes, and composition edits remain visible; filter and reveal changes are applied independently.
- Navigation grants Flow Debugger its own policy and hides Notifications unless the principal holds a role that admits a child. Route-policy coverage freezes every CentralUI route and the external Secrets route.
## Review hardening (2026-10-21)
- **Test Run bound to a deployed instance** (`BindInstanceUniqueName`, `SandboxBoundRunPolicy`) performs real site operations, so it requires the **Deployer** role (a Designer may still run an *unbound* Test Run), is confined to the caller's permitted sites (`SiteScopeService.Resolve` over the request principal; a scoped caller whose instance site cannot be resolved is refused), and writes one best-effort operator-attributed `ScriptTrace`/`RouteCall` audit row (`Extra.source = "test-run"`) via `ICentralAuditWriter`, mirroring the Flow Debugger. Denials return 403 before any worker is started.
- **Script-analysis endpoints** reject script text over 256 KB (and bodies over 1 MB) with 413 and cap concurrent requests per user at 6 (429 beyond that; `ScriptAnalysisRequestGuard`, map entries live only while a request is in flight). The `Diagnose` cache key now includes a hash of the editor context (declared parameters, sibling scripts, self attributes, children, parent), so identical code with different context never shares a cached answer. `BlockingWaitAnalyzer` advisories surface as Monaco **warning** markers (`SCADA_BLOCKING_WAIT`) on the offending line.
- **Template reads:** the Dashboard counts templates from `GetTemplateStubsAsync`; Instance Create, Template Create and Transport Export use the stub projection (adapted to childless `Template` shells); Templates and Template Edit use `GetAllTemplatesForAnalysisAsync` (no tracking, no script bodies, compositions retained). No page loads the full tracked template graph except where it edits one template (`GetTemplateWithChildrenAsync`).
- **Topology live tick** runs behind `PollGate` and a refresh never raises `_loading` (the tree stays mounted); the refreshed lists are swapped in atomically.
- **Pending-updates faults** are remembered for 10 s (`SingleFlightMemo` fault back-off) so the rail indicator's polling does not start a fleet flatten per tick against a sick database; an operator Re-check bypasses it.
- **Circuit hygiene:** `ConfigurationAuditLog` disposes its browser-time JS module; `SessionExpiry` skips the redirect after disposal; `AlarmSummary` and `TreeView` fire-and-forget calls swallow disposal/disconnect races; `FlowDebugService`'s per-circuit script cache is bounded (8 instances); the Site Event Log page retains at most 1000 rows (oldest dropped, with a note) and hints that keyword search covers at most the last 7 days unless a narrower From/To is set (server clamp `ScadaBridge:SiteEventLog:KeywordSearchMaxWindow`).
## Failover Behavior
- A **load balancer** sits in front of the central cluster and routes to the active node.
@@ -291,6 +300,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
- **Secret mirror failing** (review finding M1) — a warning badge on an online site's card when any node's `SiteHealthReport.SecretMirrorStatuses` entry lists failures, plus a compact warning panel at the top of the card body listing *node · secret: message* for each failure. The standby's failures appear here too (they ride the active node's report), so a master-key mismatch on the standby is visible before a failover exposes it. Null (no data) and all-healthy nodes render nothing; offline sites are skipped like the other warning badges.
- **Trigger failover** (`CentralFailoverControl`, Administrator only) — on the central card and on each site card. Disabled with an explanatory tooltip when there is no standby to take over. It always confirms first, and the confirmation's red button reads **Fail over** (see the dialog-label contract under Technology); the central variant's copy states that this page is served by the node being restarted and will disconnect and reconnect, while the site variant deliberately does not, since a site is a different cluster.
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
@@ -29,7 +29,7 @@ realised across two projects:
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
configures the keep-oldest split-brain resolver (`down-if-alone = on`), wires
configures the downing strategy (`auto-down` by default; `keep-oldest` with `down-if-alone` is the opt-in alternative), wires
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
@@ -197,6 +197,8 @@ Differences from central failover, all deliberate:
`akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.
`ScadaBridge:Cluster:AllowSingleNodeCluster` is **validation-only**: it merely relaxes the seed-count check (one seed instead of two) in `ClusterOptionsValidator`. It changes no runtime behaviour — not `min-nr-of-members` (always 1), not downing, not the bootstrap guard; a lone node forms its cluster because every node lists itself first in its seed list. The name is retained to avoid breaking deployed configs.
## Failure Detection Timing
Configurable defaults for heartbeat and failure detection:
@@ -218,7 +220,7 @@ If both nodes in a cluster fail simultaneously (e.g., site power outage):
2. **State recovery** (each node has its own local copy of all required data):
- **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
- **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
3. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.
3. Cold start does not involve the downing strategy: nothing is unreachable yet, so there is no pre-existing cluster to conflict with (the simultaneous-start residual is covered by the bootstrap guard above).
## Graceful Shutdown & Singleton Handover
+7 -5
View File
@@ -70,6 +70,7 @@ Both central and site clusters. Each side has communication actors that handle m
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded `GrpcInstanceStreamChannelCapacity`, default 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback, logged at Warning (first eviction, then every 500th) and exported as `scadabridge.site.stream.events_dropped` (tagged `stream=instance`), so real event loss on a debug stream is visible instead of silent. The **site-wide alarm feed has its own, larger channel** — see §6.1.
- **Bridge-session hardening (WP2.3):**
- The pre-snapshot buffer is **bounded (20 000 events, drop-oldest)** and its evictions counted (`scadabridge.central.debug_view.presnapshot_dropped`). It was previously unbounded, so a session whose snapshot never arrived grew without limit on the CENTRAL node. Dropping the oldest is correct here: the snapshot that ends the buffering phase is authoritative for anything that old.
- An **`InstanceNotFound` snapshot** (instance not deployed on the site) stops the bridge actor, and `DebugStreamService.StartStreamAsync` returns the flagged snapshot **without registering a session** — so a consumer that never calls `StopStream` (the SignalR hub only forwards the flag) leaves no registry row for the keepalive timer to Tell a dead actor.
- A **hard snapshot deadline** (`DebugStreamBridgeActor.SnapshotTimeout`, 60 s) fails the session if no `DebugViewSnapshot` arrives. Nothing else ended a session wedged in the buffering phase — a lost site reply raises no gRPC error.
- **The orphan net measures the CONSUMER, not the mailbox.** A session self-terminates after `DebugStreamBridgeActor.ConsumerIdleTimeout` (5 min) without a sign of life from its consumer, checked by a periodic self-tick (`ConsumerLivenessCheckInterval`, 30 s) against a consumer-last-seen stamp. The stamp is renewed only by `DebugStreamConsumerAlive`, which `DebugStreamService` Tells to every session still in its registry on a shared 30 s timer — holding a session there IS what "a consumer is attached" means, since both consumers (the Blazor debug view and the SignalR hub) release it on dispose/disconnect.
This replaced an Akka `SetReceiveTimeout(5 min)`. That version measured the MAILBOX, which conflates site chatter with consumer liveness: a busy site kept an abandoned session alive forever. Marking stream events `INotInfluenceReceiveTimeout` fixed that but left the timeout with nothing recurring to reset it — the snapshot lands once and the stability tick once — so **every healthy session self-terminated ~6 min in and the operator was told "Site disconnected"**. Both failure directions are pinned by tests: streaming events with keepalives survive many windows; an orphaned session terminates while events keep arriving.
@@ -100,7 +101,7 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, unpacks each frame via `ForEachEvent` (a plain frame passes through; a batch frame is flattened in order), converts protobuf messages to domain events, and invokes the `onEvent` callback.
- **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
- **SiteStreamGrpcClientFactory**: Caches `SiteStreamGrpcClient` instances keyed by `(siteId, endpoint)`. It does not read addresses itself: each streaming consumer (`SiteAlarmLiveCacheService` → `SiteAlarmAggregatorActor`, `DebugStreamService` → `DebugStreamBridgeActor`) resolves `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` row in its own DI scope (a single configured address fills both slots), not from `CentralCommunicationActor`'s cache, and flips NodeA↔NodeB itself on stream error. Before **every reopen** (fault retry, graceful-completion reopen, reconcile-tick self-heal) the consumer re-reads the site row through a `SiteStreamEndpointResolver`, so an address edited while a session or aggregator is running is dialled on its next reconnect; a lookup that fails, times out (10 s) or finds no address reopens against the endpoints already known. A client for an edited-away endpoint stays cached until site removal or process exit (documented trade-off).
- **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
#### gRPC Proto Definition
@@ -358,16 +359,17 @@ flowchart LR
Central discovers site addresses through the **configuration database**, not runtime registration:
- Each site record in the Sites table includes optional **GrpcNodeAAddress** and **GrpcNodeBAddress** fields containing the site nodes' gRPC endpoints (e.g., `http://host:8083`), used for **both** central→site command/control (`SiteCommandService`) and site→central streaming (`SiteStreamService`). The legacy **NodeAAddress** / **NodeBAddress** Akka fields are retained but no longer drive cross-cluster dialling — Akka remoting is now intra-cluster only.
- The **CentralCommunicationActor** loads all site addresses from the database at startup and builds a per-site gRPC-endpoint cache. `GrpcSiteTransport` uses it (via a `SitePairChannelProvider`) to dial each site's `SiteCommandService` as a **per-site NodeA→NodeB failover channel pair**; the **SiteStreamGrpcClientFactory** uses the same `GrpcNodeAAddress` / `GrpcNodeBAddress` for per-site streaming channels.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. Channels are rebuilt when a site's gRPC endpoints change.
- When routing a command to a site, central calls `SiteCommandService` on the site's active gRPC endpoint; on error it flips to the other node endpoint. The failover is an explicit NodeA→NodeB channel-pair flip, not internal transport routing.
- The **CentralCommunicationActor** loads all site addresses from the database at startup and builds a per-site gRPC-endpoint cache. `GrpcSiteTransport` uses it (via a `SitePairChannelProvider`) to dial each site's `SiteCommandService` as a **per-site NodeA→NodeB failover channel pair**; the streaming consumers (Alarm Summary aggregator, Debug View bridge) read the same `GrpcNodeAAddress` / `GrpcNodeBAddress` columns directly from the repository at start and again before every stream reopen (see *SiteStreamGrpcClientFactory* above), independent of this cache.
- The address cache is **refreshed every 60 seconds** and **on-demand** when site records are added, edited, or deleted via the Central UI or CLI. Command-plane channels are rebuilt when a site's gRPC endpoints change. At most **one** DB load is in flight: a tick or on-demand refresh that arrives mid-load is coalesced into a single re-run after it, each load is bounded by a deadline (`QueryTimeout`, default 30 s) so a wedged read cannot pin the refresh, and every result carries a monotonic sequence so an older result arriving after a newer one is dropped rather than re-applying stale addresses.
- A **malformed** `GrpcNodeAAddress`/`GrpcNodeBAddress` (not an absolute `http://`/`https://` URL with a host, or one the gRPC channel cannot be built from) is **rejected per node at refresh time**: `SitePairChannelProvider.UpdateSite` builds every replacement channel *before* parking the old one, so a rejected side keeps its previous channel and endpoint (the other side's valid change still applies), and the rejection is logged at **Error** naming the site and column (once per distinct bad value). `GrpcSiteTransport.ReconcileSites` reconciles each site in its own try/catch and `CentralCommunicationActor` guards the reconcile call, so one bad row can neither stop the other sites in the tick from being refreshed nor restart the actor. Write-path validation (Central UI/CLI/bundle import) is the first line of defence; this is the runtime backstop.
- When routing a command to a site, central calls `SiteCommandService` on the site's active gRPC endpoint; on error it flips to the other node endpoint. The failover is an explicit NodeA→NodeB channel-pair flip, not internal transport routing. While failed over, a background **readiness** probe returns calls to NodeA: it is a real, read-only `ExecuteLifecycle(DeploymentStateQuery)` for a sentinel instance (not a TCP/HTTP2 connect), so a node that is Up but still warming — which refuses every command with a not-dispatched `Unavailable` until `SetReady` — reads as not ready. The probe backoff (1 s → doubling → 60 s) is reset only after a **real call** on NodeA succeeds; if NodeA rejects the first calls after a probe-driven failback, the backoff keeps doubling instead of restarting at 1 s. That probation is time-boxed (5 min) and cleared by an address change: a quiet site whose NodeA fails long after the failback starts again at 1 s.
- **Heartbeats** from sites serve **health monitoring only** — they do not serve as a registration or address discovery mechanism.
- If no gRPC endpoints are configured for a site, commands to that site fail and the caller's call times out.
### Site → Central Communication
- Site nodes configure a list of **CentralGrpcEndpoints** — the gRPC h2c URLs of the central nodes (e.g., `http://scadabridge-central-a:8083`, on the central's `CentralGrpcPort`, default 8083) — replacing the former `CentralContactPoints` list of Akka addresses. A site node must list at least one; central nodes leave it empty (they host `CentralControlService`, they do not dial it). Note these are the central nodes' **direct** gRPC endpoints, not Traefik (which is HTTP/1 only).
- `GrpcCentralTransport` dials `CentralControlService` over these endpoints as a **sticky central-a→central-b failover channel pair**, and the site sends heartbeats, health reports, notification submit/status, audit/telemetry ingest, and reconcile over it.
- `GrpcCentralTransport` dials `CentralControlService` over these endpoints as a **sticky central-a→central-b failover channel pair**, and the site sends heartbeats, health reports, notification submit/status, audit/telemetry ingest, and reconcile over it. `CentralControlService.ReconcileSite` hands `CentralCommunicationActor` the gRPC call's cancellation token alongside the request (an in-process `ReconcileSiteCall` envelope), linked into `ReconcileService.ReconcileAsync`, so central stops its DB reads and pending-row staging as soon as the site's deadline expires or it disconnects.
- If the active central node goes down, the site's transport flips to the standby central endpoint transparently.
## Message Timeouts
@@ -127,7 +127,7 @@ The configuration database stores all central system data, organized by domain a
- **Bundle Unlock Attempts**: One row per permitted bundle-unlock passphrase attempt (`ClientKey`, `AttemptedAtUtc`), forming the fleet-wide sliding window behind Transport's `MaxUnlockAttemptsPerIpPerHour` throttle — see `Component-Transport.md` (register row 52). Operational history, not audit: non-partitioned, no DB-role restriction, and rows are pruned opportunistically by the application once they age past twice the window. **Denied** attempts are deliberately not recorded (a denial burns no budget), and no bundle identity is stored — the durable record of a failed unlock is the `BundleImportUnlockFailed` audit row. Central-only: written only by the Transport component, which the host registers for central roles.
### Audit Logging
- **Audit Log Entries**: Append-only audit trail (timestamp, user, action, entity type, entity ID, entity name, state as JSON). Stores only the after-state — change history is reconstructed by comparing consecutive entries. Entries are never modified or deleted. No retention policy — retained indefinitely. Indexed on timestamp, user, entity type, entity ID, and action for efficient filtering.
- **Audit Log Entries**: Append-only audit trail (timestamp, user, action, entity type, entity ID, entity name, state as JSON). Stores only the after-state — change history is reconstructed by comparing consecutive entries. Entries are never modified. Retained indefinitely by default; an optional `ScadaBridge:ConfigAudit:RetentionDays` (default `0` = off, validated `>= 0`) enables a daily, time-sliced batched purge (`ConfigAuditPurgeService`: `PurgeBatchSize` 5000 rows/batch, at most `MaxBatchesPerTick` 200 batches per daily tick, oldest first) of entries older than the window. `AfterStateJson` is capped at 256 K characters — a larger serialized state is replaced by a small valid-JSON truncation marker (`AuditTruncated`, `OriginalLength`, `StateType`, preview) so one row cannot grow without bound. The viewer query orders by `(Timestamp DESC, Id DESC)` so pages are deterministic when many rows share a timestamp (bundle imports). Indexed on timestamp, user, entity type, entity ID, and action for efficient filtering.
---
@@ -31,11 +31,12 @@ IDataConnection : IAsyncDisposable
├── ReadBatch(tagPaths) → values
├── Write(tagPath, value) → void
├── WriteBatch(values) → void
├── WriteBatchAndWait(values, flagPath, flagValue, responsePath, responseValue, timeout) → bool
├── Status → ConnectionHealth
└── Disconnected → event Action?
```
There is deliberately no write-and-wait handshake on the adapter interface: the script-facing `Attributes.WriteBatchAndWaitAsync` composes a `WriteTagBatchRequest` (batch + trigger write) with the Instance Actor's event-driven attribute waiter, so the handshake can never fall back to a polling read loop that bypasses the waiter (the adapters' former poll-based `WriteBatchAndWaitAsync` was unused and was removed — review 2026-10-21).
The `Disconnected` event is raised by an adapter when it detects an unexpected connection loss (server offline, network failure, keep-alive timeout). The `DataConnectionActor` subscribes to this event to trigger the reconnection state machine. Additional protocols can be added by implementing this interface.
### Batch Capability Seam
@@ -70,7 +71,7 @@ All protocols produce the same value tuple consumed by Instance Actors. Before t
### OPC UA
- Uses the **OPC Foundation .NET Standard Library** (`OPCFoundation.NetStandard.Opc.Ua.Client`).
- Session-based connection with endpoint discovery, certificate handling, and configurable security modes.
- Session-based connection with endpoint discovery, certificate handling, and configurable security modes. Endpoint discovery is **asynchronous** and bounded by the connection's `OperationTimeoutMs` (the discovery client is built from the connection's own `EndpointConfiguration`) and by the caller's cancellation; a cancelled discovery is never mistaken for "discovery unavailable". The `DataConnectionActor` starts every adapter `ConnectAsync` on the thread pool, so no adapter connect work — whatever runs before its first yielding await — can occupy the actor's dispatcher thread (review 2026-10-21; previously a synchronous discovery against a black-holed endpoint held the thread for the SDK's 120 s default on every reconnect tick).
- Subscriptions via OPC UA Monitored Items with data change notifications (1000ms sampling, queue size 10, discard-oldest).
- Read/Write via OPC UA Read/Write services with StatusCode-based quality mapping.
- Disconnect detection via `Session.KeepAlive` event (see Disconnect Detection Pattern below).
@@ -188,7 +189,7 @@ All settings are parsed from the data connection's configuration JSON dictionari
|-----|------|---------|-------------|
| `Endpoint` | string | `http://localhost:5000` | Gateway base URL |
| `ApiKey` | string | — | Sent to the gateway as `authorization: Bearer <key>` |
| `ClientName` | string | `scadabridge` (when blank) | MXAccess client registration name |
| `ClientName` | string | `ScadaBridge:MxGateway:ClientNamePrefix` (when blank; that option defaults to `scadabridge`) | MXAccess client registration name. The site-wide `ClientNamePrefix` is the fallback for every connection that leaves this blank — e.g. to tell sites apart on a shared gateway; it must not be blank (validated at startup). |
| `WriteUserId` | int | `0` | MXAccess user id applied to every write-back (0 = no user context) |
| `UseTls` | bool | `false` | Use TLS to a secured gateway |
| `CaFile` | string | — | Path to the CA certificate (TLS only) |
@@ -215,6 +216,7 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per-
| `SubscribeBatchDelay` | 50ms | Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there. |
| `QualityFlushInterval` | 1s | Coalescing window for pushing tag-quality counters to the health collector |
| `MxSupervisoryAdviseParallelism` | 16 | Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context |
| `MxAlarmStreamReopenDebounce` | 200ms | Window over which MxGateway alarm subscribes are coalesced into ONE reopen of the shared alarm stream (0–5 s; `0` = reopen per subscribe, the pre-2026-10-21 behaviour). Every new source still receives a fresh standing-state snapshot. |
## Subscription Management
@@ -234,7 +236,7 @@ Three subscribe paths, deliberately paced differently:
| **Reconnect re-subscribe** (`ReSubscribeAll`) | **Sequential** chunks of `SubscribeBatchSize` with `SubscribeBatchDelay` between them, all inside ONE background task; each chunk reports its per-tag results back to the actor as its own message. Replaces a tight loop that fired one fire-and-forget task per tag. |
| **Tag-resolution probe** | The whole not-in-flight unresolved set, chunked at `SubscribeBatchSize`, as one probe round per timer tick. |
Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — `SeedReadBatchSize` per chunk, `SeedReadMaxParallelism` chunks in flight, `SeedReadTimeout` per chunk — under a single `SeedOverallTimeout` deadline covering every chunk and retry round.
Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — `SeedReadBatchSize` per chunk, `SeedReadMaxParallelism` chunks in flight, `SeedReadTimeout` per chunk — under a single `SeedOverallTimeout` deadline covering every chunk and retry round. The reconnect re-seed is additionally bound to its session: entering Reconnecting, an adapter swap or actor stop cancels it, so a flapping link no longer stacks one full-tag-set seed read per flap (review 2026-10-21).
**OPC UA subscription sharding.** `RealOpcUaClient` places monitored items on the first subscription shard with free capacity (`MaxMonitoredItemsPerSubscription`), creating a shard when all are full and deleting one once it empties. A batch is N × `AddItem` plus **one** `ApplyChanges` per touched shard. Alarms & Conditions event items are pinned to a **dedicated event shard**, so `ConditionRefresh` has a single subscription id to target and `QueueSize:1000` event items never consume data-shard capacity. Every structural mutation (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation serializes behind ONE per-client gate — the SDK `Subscription` is not safe under concurrent structural mutation, and the subscribe task, resolution probe and alarm subscribe are otherwise unordered.
@@ -281,7 +283,7 @@ IAddressSpaceSearchable
Before saving or deploying an OPC UA data connection, the Central UI can ask the target site to **probe** the configured endpoint — this exercises connectivity (and TLS trust) against the live config, including edited-but-unsaved values.
- `DataConnectionManagerActor` handles `VerifyEndpointCommand` (`SiteId`, protocol, config JSON). The site spins up a **temporary** `RealOpcUaClient` from the submitted config, attempts discovery + a session with a short timeout (a few seconds), then disconnects and disposes the client. The reply is a `VerifyEndpointResult` (`Success`, a typed `FailureKind`, an error message, and an optional captured server cert).
- `DataConnectionManagerActor` handles `VerifyEndpointCommand` (`SiteId`, protocol, config JSON). The site spins up a **temporary** `RealOpcUaClient` from the submitted config, attempts discovery + a session with a short timeout (a few seconds — the budget bounds discovery too, not only the session create), then disconnects and disposes the client. The reply is a `VerifyEndpointResult` (`Success`, a typed `FailureKind`, an error message, and an optional captured server cert).
- The probe forces **`AutoAcceptUntrustedCerts = false`** and hooks the certificate-validation callback so it can **capture** an untrusted server certificate (Subject / Issuer / Thumbprint / NotBefore / NotAfter / DER) for the UI to display. Critically, the probe **never trusts** the cert — the validation callback always rejects (`Accept = false`); trusting is a separate, explicit, Admin-gated action that writes the cert into the site PKI store (see the cert-trust path in Component-SiteRuntime.md). The probe is read-only with respect to the trust boundary.
## Native Alarm Mirroring
@@ -320,7 +322,7 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and
- **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
- **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Connection-wide Mx boundaries broadcast to all subscribers while preserving their refresh epoch.
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which may increase bandwidth. Correctness also depends on the epoch-scoped snapshot cache merging concurrent live updates and rejecting stale snapshot ends (DC-5); a broad filter alone provides no such guarantee. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Restarts are **coalesced** (review 2026-10-21): subscribes arriving within `MxAlarmStreamReopenDebounce` (200 ms) of the first share ONE restart on the union prefix, so a staggered deploy batch of N alarm-bound instances costs one close/reopen and one connection-wide snapshot replay instead of N (each of which was broadcast to every already-subscribed source — O(N·M) boundary messages). A reconnect, teardown or last unsubscribe invalidates a pending restart. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which may increase bandwidth. Correctness also depends on the epoch-scoped snapshot cache merging concurrent live updates and rejecting stale snapshot ends (DC-5); a broad filter alone provides no such guarantee. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
### Protocol-Neutral Types & Messages
@@ -369,7 +371,7 @@ Each value update delivered to an Instance Actor includes:
## Connection Actor Model
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's `data_connection_definitions` table is read by the **Deployment Manager**, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's `data_connection_definitions` table is read by the **Deployment Manager**, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection; a create whose actor name still belongs to a child the manager has just **stopped** (a Remove→Create rebuild inside the old actor's dispose window — Akka reserves the name until `Terminated`) is **queued** and run on that child's `Terminated` (latest command wins; a Remove cancels it), and only a collision with a *live* child is refused (review 2026-10-21 follow-up L2) — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
>
> **The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17)**: the peer node's DCL was never "serving the old endpoint". Only the node running the `site-{SiteId}` Deployment Manager singleton ever receives a `CreateConnectionCommand`, so the standby's manager is **empty**. The divergence surfaces when the singleton's *next* incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the `Connections` snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an **arbitration rule** — the definitions store wins wherever it has an opinion, resolved *before* the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See `Component-SiteRuntime.md` → *DCL connection arbitration* and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
>
@@ -397,9 +399,11 @@ The DCL manages connection lifecycle automatically:
1. **Connection drop detection**: When a connection to a data source is lost, the DCL immediately pushes a value update with quality `bad` for **every tag subscribed on that connection**. Instance Actors see the staleness immediately and publish it to the site stream / debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive.
2. **Auto-reconnect with fixed interval**: The DCL retries the connection at a configurable fixed interval (e.g., every 5 seconds). The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table). This is consistent with the fixed-interval retry philosophy used throughout the system. Individual gRPC/OPC UA operations (reads, writes) fail immediately to the caller on error; there is no operation-level retry within the adapter.
3. **Connection state transitions**: The DCL tracks each connection's state as `connected`, `disconnected`, or `reconnecting`. All transitions are logged to Site Event Logging.
3. **Connection state transitions**: The DCL tracks each connection's state as `connected`, `disconnected`, or `reconnecting`. All transitions are logged to Site Event Logging (event type `connection`): the initial connect (`Connection established on … endpoint`, Info — added by review 2026-10-21, it previously logged nothing), connection lost / reconfigured (`Warning` / `Info`), restored on reconnect (`Info`) and failover (`Warning`).
4. **Transparent re-subscribe**: On successful reconnection, the DCL automatically re-establishes all previously active subscriptions for that connection. Instance Actors require no action — they simply see quality return to `good` as fresh values arrive from restored subscriptions.
5. **Clean client replacement on (re)connect**: On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session.
6. **Session generation on same-adapter reconnect** (review 2026-10-21, finding C1): a plain drop + reconnect reuses the adapter object, so `SwapAdapter`'s generation bump never runs. Each **reconnect attempt** therefore advances `_adapterGeneration` itself — before its `ConnectAsync` is issued — and re-hooks the adapter's `Disconnected` handler for the new generation (new handler hooked before the old one is removed). The attempt's `ConnectResult`, the new session's `Disconnected` and the `ReSubscribeAll` that follows a success all share that generation, so a disconnect queued right behind a successful result is still acted on (advancing the generation only when the result was applied would have dropped it as stale and wedged the actor Connected on a dead session — the adapter raises `Disconnected` once per session). The initial connect does not advance it: no session has existed yet. Every subscribe/alarm completion, value callback and seed started against the dead session then takes the existing stale path. That path releases a stale completion's handles against the adapter that created them, **except** when that adapter is still the current one: the handles died with the dead session, and releasing by handle against the adapter's NEW session could tear down a live item. Before the fix a dead-session initial-subscribe completion (in flight up to `SeedOverallTimeout`) was applied against the new session — parking a dead handle that made the live re-subscribe row look redundant and get released (the tag read resolved and stayed silent), or releasing a handle number the new MxGateway session had reused, and a connection-level failure row in it knocked the fresh link back into Reconnecting.
7. **Session-scoped MxGateway subscription ids** (findings C1 + L1): gateway item handles are plain ints that restart per session, so `MxGatewayDataConnection` hands out ids of the form `"{session}:{handle}"` and only ever sends a handle to the gateway when its session is the live one — a dead session's id is dropped silently instead of unadvising whatever the new session gave that number. The adapter's routing maps are cleared on every `ConnectAsync`, `DisconnectAsync` and `DisposeAsync` (they previously kept every dead session's entries, growing per flap), and a subscribe completing after a reconnect does not repopulate them. An alarm unsubscribe for an id the adapter does not know no longer decrements the shared alarm stream's reference count.
### Disconnect Detection Pattern
@@ -443,7 +447,7 @@ The DCL reports the following metrics to the Health Monitoring component via the
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
The pair is **pushed to the health collector, not pulled from it**, so it is a cached value that is only as fresh as its last push — and the collector holds it until the next one. Every site that mutates either underlying set must therefore push: entering the Connected state (a legitimate 0/0 — no instance has subscribed yet), the first-subscribe apply, the batch apply (reconnect re-subscribe and resolution probe), and unsubscribe. Missing the first-subscribe push is what made a *perfectly healthy* connection report 0/0 indefinitely — the connect-time 0/0 was never superseded, because a connection whose tags all resolve on the first attempt never reaches the batch path and has its resolution-retry timer cancelled outright for having nothing unresolved. That is the reverse of the usual staleness risk, and it blinded the diagnostic exactly when it should have read clean (deferred-work register row 55; the actor's own `GetHealthReport` reply, which really is derived on demand, was correct throughout).
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset. A value for a tag no instance subscribes to any more (a callback queued before the device unadvise landed) is dropped before the quality bookkeeping, so it can neither re-insert the tag's last-quality entry nor count a phantom bucket above the subscribed total (review 2026-10-21).
## Dependencies
@@ -465,4 +469,4 @@ The DCL reports the following metrics to the Health Monitoring component via the
OPC UA connection and endpoint verification select exactly the configured security mode. If a protected mode is unavailable, or discovery fails while protection was requested, the operation fails with the requested mode in its diagnostic; it never falls back to an unsecured session.
DCL retains each subscribed alarm source's current standing state and its in-progress snapshot. A new subscriber sharing a feed receives a scoped replay, or the already-received prefix of an in-progress frame followed by the remaining frame. Cache state is discarded on reconnect and last-owner unsubscribe. A new Mx source reopens the shared stream even when the existing prefix covers it, obtaining a fresh standing-state snapshot; callbacks from the canceled stream are ignored. Mx refresh boundaries remain connection-wide but share a distinct per-stream epoch. Both replay cache and runtime mirror ignore stale-epoch rows and apply newer live raises/clears over a concurrent refresh before committing it.
DCL retains each subscribed alarm source's current standing state and its in-progress snapshot. A new subscriber sharing a feed receives a scoped replay, or the already-received prefix of an in-progress frame followed by the remaining frame. Cache state is discarded on reconnect and last-owner unsubscribe. A new Mx source reopens the shared stream (coalesced with any other new sources inside the reopen window) even when the existing prefix covers it, obtaining a fresh standing-state snapshot; callbacks from the canceled stream are ignored. Mx refresh boundaries remain connection-wide but share a distinct per-stream epoch. Both replay cache and runtime mirror ignore stale-epoch rows and apply newer live raises/clears over a concurrent refresh before committing it. The OPC UA client's per-condition `(active, acked)` memory used to derive transition kinds follows the same removal rule as the replay cache — a condition that returns to inactive + acknowledged (the default prior state) is forgotten — so it stays bounded on servers whose condition names carry a per-occurrence component (review 2026-10-21).
@@ -167,7 +167,11 @@ one is parallel:
does not reintroduce serialization. Shape mirrors
`ArtifactDeploymentService.DeployCoreAsync`.
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
effects, write audit rows, release each operation lock.
effects, write audit rows, release each operation lock. Each finalised
instance's settled rows (staged `PendingDeployment`, snapshot, deployment
record) are then detached from the batch's shared unit of work, so the batch
does not hold N configs' JSON in the change tracker — or re-scan them on every
later save — until it ends (review 2026-10-21 04/M3).
**Disabled instances are skipped.** A bulk site deploy never touches an instance
in `Disabled` — no flatten, no send, no state change — because deploy is a legal
@@ -177,6 +181,18 @@ deliberately stopped; they are reported in the summary's `SkippedInstances` with
the reason "Disabled — bulk deploy preserves disabled state; deploy individually
to re-enable", and the single-instance deploy remains the explicit re-enable path.
**Up-to-date instances are skipped too (unless `force`), at batch cost.** The
pre-filter reads the deployed-hash projection (`GetDeployedRevisionHashesAsync`) and
the Enabled candidates' current deployment records
(`GetCurrentDeploymentStatusesAsync`) ONCE each — no snapshot `ConfigurationJson`
is materialised — and re-hashes each candidate with the hash-only flatten
(`IFlatteningPipeline.ComputeRevisionHashAsync`) on the bulk operation's **one
shared `FlattenSession`**, the same session phase 1 then prepares on, so a template
chain is walked once across both (review 2026-10-21 04/P1; previously each probe
built a private session, loaded the tracked snapshot and queried the status per
instance). The completion log reports the pre-filter's probe count, elapsed time
and chain loads alongside the totals.
**Reconciliation is bounded per batch.** Phase 1 is serial, so an unreachable site
would otherwise cost the batch one full communication-layer `QueryTimeout` (30 s)
per instance before a single deploy was sent — hours for a large site, past the
@@ -203,10 +219,50 @@ runs for it. An escape from phase 1 or phase 3 unwinds every not-yet-finalised
entry the same way: `Failed` status plus lock release, so no instance touched by
a cancelled bulk deploy can be left holding its lock.
### Area-subtree deployment (`DeployAreaAsync`, Gitea #40)
`DeployAreaAsync(areaId, user, force)` is the same bulk deploy narrowed to one
area **subtree** — the area itself plus every area nested beneath it at any
depth — surfaced as the `MgmtDeployArea` management command, the CLI
`deploy area --id`, and the Central UI Topology page's **Deploy all…** context
item on area nodes (the site node's item calls `DeploySiteAsync`). Both entry
points run one private core (`DeployInstancesOfSiteAsync`) that loads the site's
instances, keeps only those whose `AreaId` is in the subtree (area-less instances
belong to the whole-site scope only), and then applies everything above
unchanged: the Disabled skip (even under `force`), the up-to-date skip unless
`force`, the three phases, not-all-or-nothing, and the same
`SiteDeploymentSummary` with `SkippedInstances`. An unknown area is a failure
result, not an empty success.
The subtree is resolved from ONE read of the site's areas
(`GetAreasBySiteIdAsync`) walked in memory over `ParentAreaId`; each area is
visited at most once, so a corrupt parent chain that loops cannot hang the walk.
`PreviewSubtreeDeployAsync(siteId, areaId?)` shares that walk to return the
confirm dialog's `SubtreeDeployPreview(Total, Disabled)` counts (`areaId` null =
the whole site; an area of another site is a failure) — a plain DB read with no
staleness probe, so the up-to-date count is only known once the deploy reports
its skipped rows.
`DeployInstanceAsync` is composed from the same three phase helpers with a batch of
one, so the two entry points cannot drift on deployment identity, idempotency, lock
coverage, or optimistic concurrency.
### Startup-reconcile staging
A site node that missed a deploy reports its inventory on startup and central
(`Communication/ReconcileService`) stages fresh `PendingDeployment` fetch tokens for
the gap. The pass costs a **fixed number of round trips whatever the gap size**
(review 2026-10-21 03/P-1): the expected-set projection (no config JSON), ONE
untracked read of the gap instances' snapshot configs
(`GetDeployedSnapshotsByInstanceIdsAsync`), and ONE batched insert-if-absent
(`StagePendingBatchIfAbsentAsync` — expired rows deleted set-based, live rows from an
in-flight deploy or the peer node's concurrent reconcile handed back instead of
superseded, one `SaveChanges`; a failed batch save falls back to per-row staging so
one bad row costs only itself). Central's work is bounded by `QueryTimeout`, the
site's own deadline for the reply. On the site, `SiteReconciliationActor` retries a
failed or partially-failed pass up to **3** times with exponential back-off (30 s,
60 s) instead of waiting for the next node restart.
### Terminal deployment-record retention
Deployment records are insert-only — one row per deploy attempt — so without a
@@ -221,6 +277,11 @@ never purged regardless of age: it is precisely the row the query-before-redeplo
reconciliation reads to decide whether a prior deploy actually landed at the site,
and expiring it by age would silently disable that idempotency guard.
The same window also bounds **`SystemArtifactDeploymentRecords`** — one row per
*Deploy Artifacts* run, written after the sweep and never read back — which the
sweep deletes by `DeployedAt` in the same bounded batches. Before 2026-10 that table
had no purge path at all.
## Diff View
Before deploying, the Deployment Manager can request a diff from the Template Engine showing:
@@ -238,10 +299,8 @@ The system maintains two views per instance:
These are compared to determine staleness and generate diffs.
The comparison path is **hash-only by design**. When the Template-Derived
Configuration is re-flattened for a comparison (the Deployments comparison page)
or for the Transport stale-instance probe
(`IStaleInstanceProbe.GetCurrentRevisionHashAsync`, used per-instance across a
bundle import), the pipeline is invoked with `validateScripts: false`, which
Configuration is re-flattened for a comparison (the Deployments comparison page),
the pipeline is invoked with `validateScripts: false`, which
skips **only** the expensive Roslyn script-compilation stage
(`ValidationService.ValidateScriptCompilation` — a non-collectible assembly load
per script). Structural and semantic validation still run. These read-only paths
@@ -385,16 +444,25 @@ out below.
check 5). An old site ignores proto field 9, so the empty list costs it
nothing, and the hash is unaffected (empty == null above).
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
written per site deploy that carried secrets or a missing-secret failure
(never for a site with no secret references at all) — after-state is `{
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
`RetryForSiteAsync`, including the fail-closed path (audited even though
nothing was sent). The row carries `secrets` and `missing` only — there is
**no `removed` field** (design §3.7 named one), and a deploy that only
removes secrets (the site's last reference dropped, an empty list shipped)
writes **no central row**; removals are logged on the site by
`DeployedSecretsMirror`.
written per site deploy that carried secrets, failed on a missing secret or a
secret-store read, or retired previously-deployed secrets (never for a site
with no secret involvement at all) — after-state is `{ deploymentId,
secrets: [{name, revision}], missing: [name], removed: [name], success,
error }`, which structurally excludes ciphertext. Written on both
`DeployCoreAsync` and `RetryForSiteAsync`, including the fail-closed paths
(audited even though nothing was sent). `removed` is the site's reply field
`ArtifactDeploymentResponse.SecretsRemoved` — the `deployed_secrets` rows that
deploy soft-removed — so a deploy that only drops the site's last secret
reference is still audited centrally. A site that predates the field never
reports it.
- **Central secret-store outage.** `SiteSecretsCollector` returns a store read
failure as `StoreError` rather than throwing, so it is contained per site: a
site that references a secret fails (`Central secret store could not be read
while resolving '<name>' (...)`, nothing sent, not stamped), every other site
deploys, and `ComputeCurrentArtifactHashesAsync` omits only the affected sites
from its map (they read *Out of date* on the Sites page) instead of failing the
fleet-wide pending-updates check. A site with no secret references never
touches the store.
- **Secret rotation and live connections.** The site writes and mirrors the
shipped secrets before it pushes data connections to the DCL, so a
connection created by the same deploy resolves a new secret on first
@@ -434,12 +502,17 @@ cadence, agreeing exactly with the per-instance Topology badge
- All the flattens share **one `FlattenSession`** for the batch, so a template
chain used by N instances is walked once — the same trick `DeploySiteAsync`
uses.
uses — and each is **hash-only** (`IFlatteningPipeline.ComputeRevisionHashAsync`):
no structural/semantic validation and none of the shared-script / schema-library
loads that feed it, since staleness only ever compares the hash (identical to
`FlattenAndValidateAsync`'s `RevisionHash`). `StaleInstanceProbe` uses the same
entry point.
- The deployed side is read as the `(InstanceId, RevisionHash)` **projection**
(`IDeploymentManagerRepository.GetDeployedRevisionHashesAsync`), never the fat
`ConfigurationJson` for the whole fleet.
- **Mismatch-only normalisation**: only when the projected hash disagrees with
the re-flatten is the full `DeployedConfigSnapshot` read and the I-1 List-value
the re-flatten is the full `DeployedConfigSnapshot` read — in **one untracked
batch** for every mismatched instance — and the I-1 List-value
normalisation applied (see *Deployed vs. Template-Derived State* above) — the
same normalisation `GetDeploymentComparisonAsync` applies, so the aggregate
never disagrees with the Topology badge on an old-form List snapshot, without
@@ -461,7 +534,11 @@ cadence, agreeing exactly with the per-instance Topology badge
`ArtifactsDeployedAt`, `DeployedInstanceCount`, `StaleInstanceCount`,
`UnknownInstanceCount`) and fleet totals `StaleInstanceCount` /
`StaleArtifactSiteCount`. Only instances with a deployed snapshot (`State !=
NotDeployed`, the same gate Topology uses) are evaluated. **`NeverDeployed` is
NotDeployed`, the same gate Topology uses) are evaluated; the instance list is a
slim untracked projection (`GetDeployedInstanceSummariesAsync`: id, site, state —
no override collections), and `ComputeCurrentArtifactHashesAsync` reads each
referenced secret from the central store once per evaluation, however many sites
reference it. **`NeverDeployed` is
deliberately not a pending update** — a freshly-created site the operator has not
gotten to yet has nothing drifted, so it is surfaced on the Sites page but never
counted toward the fleet totals or the rail indicator's badge count. Nothing in
@@ -512,6 +589,6 @@ The Deployment Manager sends the following commands to sites via the Communicati
### Authoritative staleness and bulk deadlines
`StaleInstanceProbe` no longer memoizes revision hashes against process-local watermarks. Every probe starts a fresh flatten session, so a configuration edit through another central is visible to the next deployment skip decision. This trades extra flattening work for correctness across shared-database writers.
`StaleInstanceProbe` no longer memoizes revision hashes against process-local watermarks. Every probe starts a fresh flatten session, so a configuration edit through another central is visible to the next deployment skip decision. This trades extra flattening work for correctness across shared-database writers. The probe (used per instance across a bundle import), the bulk-deploy pre-filter and the pending-updates sweep all use the hash-only `IFlatteningPipeline.ComputeRevisionHashAsync` — flatten + hash, no validation — whose hash equals `FlattenAndValidateAsync`'s, so all three agree with the deploy gate on what "stale" means. The bulk pre-filter shares its operation's session (a within-operation share, not the removed cross-operation memo).
Management bulk deployment uses `LongRunningCommandTimeout` as an overall scheduling/send deadline. Cancellation stops scheduling more instances, fails unfinished eligible rows, and releases prepared operation locks. Completed and deliberately skipped outcomes remain in the response. HTTP Ask reserves an additional60 seconds for cleanup and response; the bulk CLI defers its timeout to this server-owned deadline.
@@ -120,6 +120,8 @@ Scripts choose between two call modes per invocation, mirroring the dual-mode da
- Database connections use **standard ADO.NET connection pooling** per named connection. No custom pool management.
- Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed.
- **Definition cache (central only).** `ExternalDefinitionLookupCache` holds resolved external-system, method and database-connection definitions for `ExternalSystemGateway:DefinitionCacheTtl` (default 0 = off in the options; the central host sets 10 s because each lookup there is a SQL Server round-trip). Only found definitions are cached, so a new definition is visible immediately; an edit/delete within one TTL. Sites keep it off — `SiteExternalSystemRepository` already serves external systems from the deploy-invalidated `ExternalSystemDefinitionCache` snapshot.
- GET/DELETE parameters are rendered into the query string with scalars via `ToString()` and collections/objects JSON-serialised (never a CLR type name).
- Synchronous failures on `Database.Connection()` (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling.
## Dependencies
@@ -39,6 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. |
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
## Reporting Protocol
@@ -232,6 +233,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
- **Communication Layer**: Transports health reports from sites to central.
- **Data Connection Layer (site)**: Provides connection health metrics.
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
- **Cluster Infrastructure (site)**: Provides node role status.
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
+10 -3
View File
@@ -47,7 +47,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
|---------|--------------|-------|----------|
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site: hosts `SiteCommandService` + `SiteStreamGrpcServer`, default 8083), CentralGrpcPort (central: hosts `CentralControlService`, default 8083) |
| `ScadaBridge:Cluster` | `ClusterOptions` | ClusterInfrastructure | SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers |
| `ScadaBridge:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb, MachineDataDb connection strings; Site: SQLite paths |
| `ScadaBridge:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb connection string; Site: SQLite paths |
#### Per-Component Sections
@@ -58,7 +58,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
| `ScadaBridge:HealthMonitoring` | `HealthMonitoringOptions` | Health Monitoring | ReportInterval, OfflineTimeout |
| `ScadaBridge:SiteEventLog` | `SiteEventLogOptions` | Site Event Logging | RetentionDays, MaxStorageMb, PurgeScheduleCron |
| `ScadaBridge:Communication` | `CommunicationOptions` | Communication | DeploymentTimeout, LifecycleTimeout, QueryTimeout, TransportHeartbeatInterval, TransportFailureThreshold |
| `ScadaBridge:Security` | `SecurityOptions` | Security & Auth | LdapServer, LdapPort, LdapUseTls, JwtSigningKey, JwtExpiryMinutes, IdleTimeoutMinutes |
| `ScadaBridge:Security` | `SecurityOptions` | Security & Auth | LdapServer, LdapPort, LdapUseTls, JwtSigningKey (HS256 key for the CLI/API bearer token from `/auth/token`; >= 32 bytes, validated on Central), JwtExpiryMinutes (bearer lifetime), IdleTimeoutMinutes (cookie idle window), RoleRefreshThresholdMinutes, RoleMappingCacheSeconds |
| `ScadaBridge:InboundApi` | `InboundApiOptions` | Inbound API | DefaultMethodTimeout, MaxRequestBodyBytes, ApiMethodCacheTtl, StandbyForwardPeerEndpoints (HTTP base URLs of the other central node — a standby relays `POST /api/*` there instead of answering 503), StandbyForwardTimeout |
| `ScadaBridge:Notification` | `NotificationOptions` | Notification Service | (SMTP config is stored in the central config DB, not in appsettings) |
| `ScadaBridge:NotificationOutbox` | `NotificationOutboxOptions` | Notification Outbox | Dispatcher poll interval, stuck-age threshold, retention window (delivery retry settings reuse the central SMTP configuration) |
@@ -83,7 +83,7 @@ Before the Akka.NET actor system is created, the Host must validate all required
- `NodeConfiguration.RemotingPort` must be in valid port range (1–65535).
- Site nodes must have `GrpcPort` in valid port range (1–65535) and different from `RemotingPort`.
- Site nodes must have a non-empty `SiteId`.
- Central nodes must have non-empty `ConfigurationDb` and `MachineDataDb` connection strings.
- Central nodes must have a non-empty `ConfigurationDb` connection string.
- Site nodes must have non-empty SQLite path values. Site nodes do **not** require a `ConfigurationDb` connection string — all configuration is received via artifact deployment and read from local SQLite.
- At least two seed nodes must be configured.
@@ -345,3 +345,10 @@ built)* section for where the shipped code deviates from the original design).
- **CentralUI / InboundAPI**: The Host maps their web endpoints into the ASP.NET Core pipeline on central nodes.
- **ManagementService**: The Host registers the ManagementActor on central nodes; the CLI reaches it over the HTTP `/management` endpoints (in-process via `ManagementActorHolder`), not via any cluster transport. No ClusterClientReceptionist is configured (removed in Phase 4).
- **HealthMonitoring**: The Host's startup validation and logging configuration provide the foundation for health reporting.
## Review 2026-10-21 notes (Host edge)
- **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node).
- **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
- **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
- Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.
+2 -2
View File
@@ -175,7 +175,7 @@ The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a
## Script Compilation & Hot-Reload
API method scripts are compiled at central startup — all method definitions are loaded from the configuration database and compiled into in-memory delegates.
API method scripts are pre-compiled in the **background after startup** by `InboundMethodWarmupService` (a hosted service registered after `AkkaHostedService`, so it never delays the cluster join or `/health/*`): it waits for `ApplicationStarted`, then — only once this node is the active central node (`IActiveNodeGate`; a standby pays nothing until a failover promotes it) — loads all method definitions and compiles them into in-memory delegates. Lazy compile + the per-request revision check guarantee correctness regardless. Compiles are **single-flight per method name + script revision**: N concurrent first callers share one compile (and one non-collectible loaded assembly). `InboundScriptExecutor.CompileAndRegister(ApiMethod, out IReadOnlyList<string> errors)` compiles once and returns the diagnostics, so a save needs no separate validate pass; the management Create/Update path uses its sibling `CompileForPublish(method, out errors, out publish)`, which compiles once for the `compileWarning` and hands back a `publish` action queued as the post-commit registration — one compile (one pinned assembly) per save even inside a configuration transaction (review 2026-10-21 05/M1). `ApiMethodCache` prunes expired entries on a miss once large (scopes are exact method names — pinned by test — so growth is bounded).
### Update Workflow
@@ -367,7 +367,7 @@ A standby now **relays** the request to the active peer instead (`InboundApiEndp
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the total relay budget and should exceed the longest method timeout plus network overhead. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — the options snapshot is read per relay.
- **Pure relay.** Method, path, query string, body and credentials travel unchanged; hop-by-hop/Host/forwarding-provenance headers are excluded and authenticated provenance is rebuilt, so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The serving node owns its `ApiInbound` row; the originating standby retains a failure row when no peer serves or the outcome is uncertain. Verified internal standby refusals are not duplicated.
- **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
- **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. A relaying standby skips the audit middleware's request/response capture entirely; the body is read once, by the forwarder (bounded by `MaxRequestBodyBytes`), so it can be re-sent to a second peer. On the node that executes, request bodies up to the 1 MiB cap are buffered in memory (`EnableBuffering` with the cap as threshold), not spilled to a temp file. The relay proof (`X-ScadaBridge-Relay`) carries a random nonce, is signed fresh per peer attempt, and is **single-use** within its 300 s window: the receiver keeps a bounded `(ExecutionId, nonce)` set and treats a repeat (or a nonce-less proof from an older peer) as unverified — the request still runs, but a captured proof cannot re-stamp ExecutionId/IP.
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, proven name-resolution failure, or connection refusal permits trying the next peer. Timeout or ambiguous response loss returns 502 `OUTCOME_UNKNOWN` without replay. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
- **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: <relaying host>`. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
- **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.
+2 -2
View File
@@ -90,9 +90,9 @@ A timer fires every `SampleInterval` (default 60s; a first tick after five secon
Each source has a `SourceTimeout` (default 10 seconds) bounding the actual wait as well as requesting cancellation. A source that ignores cancellation keeps its task, cancellation source, and DI scope until it actually finishes; later ticks skip that registration instead of starting replacements. Healthy sources continue with separate scopes, so they do not share a busy `DbContext` with unfinished work. A root-DI singleton admits at most one unfinished collection per registration across recorder instances, including actor stop/recreation on the same provider. Admission is atomic; replacing an actor does not start another call against a still-running source. Different root providers have independent admission state.
**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. During a **failover overlap** — the old singleton incarnation's in-flight fold and the new node's first fold both `Add`-ing the same `(series, hour)` row — the loser of the unique `IX_KpiRollupHourly_Series` upsert race discards its **whole fold pass** (a single `SaveChanges`), logged at Information and repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the **pass, not the row** (arch-review 04 round 2, R5).
**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. During a **failover overlap** — the old singleton incarnation's in-flight fold and the new node's first fold both `Add`-ing the same `(series, hour)` row — the loser of the unique `IX_KpiRollupHourly_Series` upsert race discards its **whole fold pass** (a single `SaveChanges`), logged at Information and repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the **pass, not the row** (arch-review 04 round 2, R5). Only a unique-key violation (SQL Server 2601/2627; the SQLite `UNIQUE constraint failed` fault on the test provider) is classified as that benign race — any other save failure propagates and the recorder logs it at Error, so a fold that faults for a real reason is no longer invisible.
**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (`kpi-rollup-backfill` timer) that folds the retention window, reusing the same `FoldHourlyRollupsAsync` path. The `FoldHourlyRollupsAsync` repository call materializes its whole window in memory, so the backfill folds the window in **≤24 h slices, oldest-first** — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds **interleave** instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair, so the two idempotent upserts never race on overlapping rows. On a **failover restart** the plan first consults the `IKpiHistoryRepository.GetLatestRollupHourAsync` watermark: if the newest folded hour is already within `RollupLookbackHours` of now (the common case) the whole historical pass is **skipped** — the periodic fold's lookback covers the tail — otherwise the window is **shrunk to the un-rolled tail** `[watermark, TruncateToHour(now))` (re-folding the watermark hour itself as an idempotent safety margin) instead of the full `RetentionDays` floor. A `_backfillDone` latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.
**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (`kpi-rollup-backfill` timer) that folds the retention window, reusing the same `FoldHourlyRollupsAsync` path. The `FoldHourlyRollupsAsync` repository call reads raw samples **one hour at a time** (an untracked projection of a single hour, grouped in memory by series), so resident memory is bounded by one hour of samples regardless of the window; the backfill still folds the window in **≤24 h slices, oldest-first** — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds **interleave** instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair, so the two idempotent upserts never race on overlapping rows. On a **failover restart** the plan first consults the `IKpiHistoryRepository.GetLatestRollupHourAsync` watermark: if the newest folded hour is already within `RollupLookbackHours` of now (the common case) the whole historical pass is **skipped** — the periodic fold's lookback covers the tail — otherwise the window is **shrunk to the un-rolled tail** `[watermark, TruncateToHour(now))` (re-folding the watermark hour itself as an idempotent safety margin) instead of the full `RetentionDays` floor. A `_backfillDone` latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.
**Retention.** A daily purge timer (`PurgeInterval`, default 24h) now runs **both** purges: raw `KpiSample` rows older than `RetentionDays` (default 90) via `IKpiHistoryRepository.PurgeOlderThanAsync`, **and** hourly rollup rows older than `RollupRetentionDays` (default 365) via `IKpiHistoryRepository.PurgeRollupsOlderThanAsync` (a one-hour-sliced batched DELETE mirroring the raw purge). Rollups deliberately **outlive** raw samples so long-range trend charts have data after the raw rows are purged; the options validator enforces `RollupRetentionDays >= RetentionDays` so a window never falls into a hole where raw is purged but no rollup exists.
@@ -67,6 +67,9 @@ Content-Type: application/json
- `ManagementUnauthorized` → HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`
- Authentication failure → HTTP 401
- Actor timeout → HTTP 504
- Large request while another is in flight → HTTP 409 with `{ "error": "...", "code": "IMPORT_IN_PROGRESS" }` (see below)
**Large-request single-flight and streaming parse (review 2026-10-21 05/M2).** The body cap is raised to 200 MB for bundle import/preview. A request whose `Content-Length` exceeds 16 MB — or is unknown (chunked) — must take a per-node `LargeRequestGate` with `Wait(0)` *after authentication and before the body is read*, and holds it until the actor replies; a second large request is refused with **409 `IMPORT_IN_PROGRESS`** (retry when the first completes) rather than materialising another copy and then waiting at the actor's `TransportGate`. Ordinary (small) commands never touch the gate. The command is parsed straight off the request stream (`ParseCommandAsync`: one UTF-8 buffer, the payload deserialised directly into the command type once `command` has been read — the CLI sends `command` first; a payload that precedes it is buffered as a `JsonElement`), replacing the old `ReadToEnd` string → `JsonDocument` → `GetRawText` → `Deserialize` chain (~4 copies, ~1.1 GB peak for a 200 MB body). Error messages (`Invalid JSON body.`, `Missing 'command' field.`, `Unknown command: '…'.`, `Failed to deserialize payload: …`) are unchanged.
The endpoint performs LDAP authentication and role resolution server-side, collapsing the CLI's previous two-step flow (ResolveRoles + actual command) into a single HTTP round-trip.
@@ -110,17 +113,18 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
- **AddTemplateAttribute** / **UpdateTemplateAttribute** / **DeleteTemplateAttribute**: Manage attributes on a template.
- **AddTemplateAlarm** / **UpdateTemplateAlarm** / **DeleteTemplateAlarm**: Manage alarm definitions on a template.
- **AddTemplateNativeAlarmSource** / **UpdateTemplateNativeAlarmSource** / **DeleteTemplateNativeAlarmSource**: Manage native alarm source bindings (the read-only OPC UA Alarms & Conditions / MxAccess Gateway mirror) on a template — `Name`, `ConnectionName`, `SourceReference`, optional `ConditionFilter` / `Description`, `IsLocked`. Mutations gated to the **Design** role; handlers write through `ITemplateEngineRepository` directly.
- **AddTemplateNativeAlarmSource** / **UpdateTemplateNativeAlarmSource** / **DeleteTemplateNativeAlarmSource**: Manage native alarm source bindings (the read-only OPC UA Alarms & Conditions / MxAccess Gateway mirror) on a template — `Name`, `ConnectionName`, `SourceReference`, optional `ConditionFilter` / `Description`, `IsLocked`. Mutations gated to the **Design** role; handlers write through `ITemplateEngineRepository` directly — so, unlike scripts/alarms/composition slots (checked by `TemplateService`), Add/Update validate `Name` with `ActorNameRules` themselves and reject an illegal Akka actor-name element (space, `/`, leading `$`, …) with a readable error, because the name becomes part of the site actor name `native-alarm-{canonical}` (review 2026-10-21, follow-up to 01/SR-C1).
- **ListTemplateNativeAlarmSources** (`TemplateId`): List a template's native alarm source bindings (read-only; any authenticated user).
- **AddTemplateScript** / **UpdateTemplateScript** / **DeleteTemplateScript**: Manage scripts on a template.
- **AddTemplateComposition** / **DeleteTemplateComposition**: Manage feature module compositions on a template.
### Instances
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`ExportBundle` still loads the full table internally — a logged scale follow-up, arch-review P2 deferred. `QueryDeployments` was closed by WP2.5, below.)
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2); same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. Paging, the site-scope filter and search all run **in SQL** (`ICentralUiRepository.GetInstancesPagedAsync`, review 2026-10-21 08/P2): a site-scoped caller's `PermittedSiteIds` become the query's site filter (intersected with an explicit `SiteId`; a non-integer grant is dropped, which can only narrow), so a page can never surface an out-of-scope instance. **`SearchTerm` is a unique-name prefix match (`StartsWith`)**, not a substring match — the previous leading-wildcard `LIKE '%term%'` could not use `IX_Instances_UniqueName`. The response is still a bare array (the total count is not returned). (`ExportBundle` still loads the full table internally — a logged scale follow-up, arch-review P2 deferred. `QueryDeployments` was closed by WP2.5, below.)
- **CreateInstance**: Create a new instance from a template.
- **UpdateInstanceOverrides** (`SetInstanceOverrides`): Set attribute overrides on an instance. Every requested attribute is validated against the instance's template up front (unknown or locked ⇒ the whole batch is rejected before any write), then the batch is applied as **one bulk read of the existing override rows plus one commit** (WP2.5). It previously ran K independent read-modify-commit cycles — each re-reading the instance, the template's attributes and the existing overrides, then committing and writing its own audit row — so the all-or-nothing promise was only as good as the pre-validation. It is now a real single-transaction apply, with one audit row summarising the batch.
- **MgmtDeploySite** (`SiteId`, WP2.5): Bulk-deploy every deployable instance at one site. Site scope is enforced on the SITE (the command's target); every instance the batch touches belongs to that site by construction. Same `Deployer` authority as `MgmtDeployInstance` — it is N of those, not a new class of privilege — and registered as a long-running command so the caller does not time out mid-batch. Semantics, phasing and failure policy: see `Component-DeploymentManager.md` → Bulk site deployment.
- **MgmtDeployArea** (`AreaId`, `Force`, Gitea #40): Bulk-deploy every deployable instance in one area **subtree** (the area plus all descendant areas) with the site deploy's rules — Disabled always skipped, up-to-date skipped unless `Force`, not all-or-nothing — answering with the same `SiteDeploymentSummary`. Site scope is enforced on the area's owning site; an area that does not resolve is denied for a site-scoped caller exactly like an out-of-scope one (no existence oracle) and is a plain "not found" error for a system-wide caller. `Deployer`-only, long-running with the same overall deadline, `BULK_DEPLOY_TIMEOUT` mapping and +1 min Ask grace as `MgmtDeploySite`. See `Component-DeploymentManager.md` → Area-subtree deployment.
- **SetInstanceAlarmOverride** / **DeleteInstanceAlarmOverride** / **ListInstanceAlarmOverrides**: Manage per-instance computed-alarm overrides.
- **SetInstanceNativeAlarmSourceOverride** / **DeleteInstanceNativeAlarmSourceOverride**: Retarget or clear a per-instance native alarm source binding, keyed by `SourceCanonicalName` — `ConnectionNameOverride` / `SourceReferenceOverride` / `ConditionFilterOverride` each apply only when non-null. Gated to the **Deployment** role.
- **ListInstanceNativeAlarmSourceOverrides** (`InstanceId`): List an instance's native alarm source overrides (read-only).
@@ -131,7 +135,7 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
### Sites
- **ListSites** / **GetSite**: Query site definitions.
- **CreateSite** / **UpdateSite** / **DeleteSite**: Manage site definitions.
- **CreateSite** / **UpdateSite** / **DeleteSite**: Manage site definitions. Create/Update validate `GrpcNodeAAddress` / `GrpcNodeBAddress` at the write boundary with `GrpcEndpointRules.TryValidate` (both optional; a non-empty value must be an absolute `http://` or `https://` URL with a host, e.g. `http://host:8083`) and reject a malformed value with a readable `COMMAND_FAILED` error naming the field — a scheme-less address used to be persisted and then throw inside central's channel build on the next site-address refresh (review 2026-10-21 03/R-1).
- **ListAreas** / **CreateArea** / **UpdateArea** / **DeleteArea**: Manage area hierarchies per site.
- **MoveArea** (`AreaId`, `NewParentAreaId?`): Re-parent an area within its own site; a null `NewParentAreaId` moves it to the site root. Delegates every rule to `AreaService.MoveAreaAsync` — rejects self-parent, descendant-parent (cycle), cross-site parent, and sibling name collision at the target level; the service writes the `"Move"` audit row. Same any-of `[Designer, Deployer]` gate as the other area mutations. CLI: `site area move`.
@@ -148,6 +152,7 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
- **DeployInstance**: Deploy configuration to a specific instance (includes pre-deployment validation).
- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site. The command's `SiteId` is honored (arch-review C2): a value routes to the single-site deploy path (`ArtifactDeploymentService.DeployToSiteAsync`), while `null` deploys fleet-wide (`DeployToAllSitesAsync`). Site scope is enforced — a site-scoped (non-Administrator) Deployer may only target a site within its `PermittedSiteIds`, and may **not** deploy fleet-wide (`SiteId is null` is rejected with `SiteScopeViolationException` → `ManagementUnauthorized`, since `EnforceSiteScope(null)` is a deliberate no-op); fleet-wide deployment requires a system-wide Deployer or Administrator.
- **DeploySite** (`MgmtDeploySite`, WP2.5): Bulk-deploy every deployable instance at one site — see the Instances section above and `Component-DeploymentManager.md`.
- **DeployArea** (`MgmtDeployArea`, Gitea #40): Bulk-deploy every deployable instance in one area subtree — see the Instances section above.
- **GetDeploymentStatus**: Query deployment status.
- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentQueryResult` envelope — `{ deployments, totalCount, page, pageSize }` over `DeploymentListRow`. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`.
@@ -169,6 +174,8 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
**Server-side pending TTL (arch-review S2).** A `Pending` secured write is not valid indefinitely. Both `ApproveSecuredWriteCommand` and `RejectSecuredWriteCommand` call a shared `TryExpireIfStaleAsync` helper immediately after the `Pending`-status check: if the row's `SubmittedAtUtc` is older than the configured `SecuredWritePendingTtl` (`ManagementServiceOptions`, default **24 h**), the helper atomically transitions it `Pending → Expired` via `ISecuredWriteRepository.TryMarkExpiredAsync` (compare-and-swap — no verifier recorded, `VerifierUser` stays `NULL`), emits a best-effort `SecuredWriteExpire` audit row (system actor, `Discarded` status — the closest terminal `AuditStatus`), and the command fails with an "expired" error. A stale setpoint therefore can **never** be approved or executed days later, and no `WriteTagRequest` is relayed. A non-positive TTL disables expiry. `ListSecuredWritesCommand` additionally runs an **opportunistic sweep**: on every read it walks the returned page and calls the same helper for each `Pending` row, so overdue writes surface as `Expired` the next time anyone views the queue — no dedicated expiry-timer singleton is needed because the ManagementActor already runs on every central node and the CAS makes concurrent multi-node sweeps idempotent.
**Terminal-row purge (review 2026-10-21 08/G3).** The same list sweep also purges decided/executed rows — `Approved`, `Rejected`, `Expired`, `Executed`, `Failed` — whose last lifecycle instant (`ExecutedAtUtc`, else `DecidedAtUtc`, else `SubmittedAtUtc`) is older than `SecuredWriteTerminalRetention` (default **90 days**), via `ISecuredWriteRepository.PurgeTerminalAsync` (bounded set-based delete, 1 000 rows per pass, so a backlog converges over successive passes). `Pending` rows are never purged — they expire first. The pass is throttled per node to once per `SecuredWriteTerminalPurgeInterval` (default 1 h), is idempotent across both central nodes, and is best-effort (a failure is logged and never fails the list). `PendingSecuredWrites` is operational state; the append-only `AuditLog` `SecuredWrite*` rows remain the lifecycle history after a purge. Previously no component deleted rows, so the table and the list's offset-paged scan grew forever.
### External Systems
- **ListExternalSystems** / **GetExternalSystem**: Query external system definitions.
@@ -187,6 +194,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
- **ListApiKeys** / **CreateApiKey** / **UpdateApiKey** / **EnableApiKey** / **DisableApiKey** / **DeleteApiKey**: Manage API keys.
- **ListRoleMappings** / **CreateRoleMapping** / **UpdateRoleMapping** / **DeleteRoleMapping**: Manage LDAP group-to-role mappings.
- **ListScopeRules** / **AddScopeRule** / **DeleteScopeRule**: Manage site scope rules on role mappings.
- Every role-mapping and scope-rule mutation (Create/Update/DeleteRoleMapping, Add/DeleteScopeRule) calls `RoleMappingCache.Invalidate()` as a **post-commit** action, so the change takes effect for the next role mapping on the node that served the command instead of after the `RoleMappingCacheSeconds` TTL (default 30 s); the other central node still converges within that TTL. Post-commit, because invalidating inside the transaction would let a concurrent mapping re-cache the pre-change table.
### Audit Log
@@ -213,7 +221,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
### Inbound API Methods
- **ListApiMethods** / **GetApiMethod**: Query inbound API method definitions.
- **CreateApiMethod** / **UpdateApiMethod** / **DeleteApiMethod**: Manage inbound API methods.
- **CreateApiMethod** / **UpdateApiMethod** / **DeleteApiMethod**: Manage inbound API methods. Create/Update compile the script **once** (`InboundScriptExecutor.CompileForPublish`): the diagnostics become the result's `compileWarning` and the handler registration is queued as a post-commit action reusing that compile — each compile pins a non-collectible assembly, and the old validate-then-register pair compiled twice per save (review 2026-10-21 05/M1).
### Health
@@ -297,8 +305,10 @@ The ManagementActor receives the following services and repositories via DI (inj
| Section | Options Class | Contents |
|---------|--------------|----------|
| `ScadaBridge:ManagementService` | `ManagementServiceOptions` | `CommandTimeout` (`TimeSpan`, default 30 s) — Ask timeout the HTTP endpoint applies when forwarding to the `ManagementActor`. A non-positive configured value falls back to the 30 s default. |
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`, `MgmtDeploySite`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`, `MgmtDeploySite`, `MgmtDeployArea`; the two bulk deploys get one extra minute of Ask grace beyond it); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
| | | `SecuredWritePendingTtl` (`TimeSpan`, default 24 h) — age after which a `Pending` secured write is transitioned to `Expired` and can no longer be approved/executed; enforced at approve/reject and swept opportunistically on list. A non-positive value disables expiry (arch-review S2). |
| | | `SecuredWriteTerminalRetention` (`TimeSpan`, default 90 days) — age (from the row's last lifecycle instant) after which an `Approved`/`Rejected`/`Expired`/`Executed`/`Failed` secured-write row is purged on the list sweep. A non-positive value disables the purge (review 2026-10-21 08/G3). |
| | | `SecuredWriteTerminalPurgeInterval` (`TimeSpan`, default 1 h, must be ≥ 0) — minimum interval between purge passes on one node; zero purges on every list. |
## Dependencies
@@ -115,10 +115,11 @@ Delivery has no row-level claim or lease: dispatchers read due rows and persist
The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutboxActor`:
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed — capped at `DispatchBatchSize` (default 100). The batch is **shared between the two statuses**: due `Retrying` rows take at most half of it and `Pending` rows the rest, each oldest-first, with a share the other status cannot fill back-filled. A single oldest-first claim used to let a retrying backlog (always older than fresh traffic) monopolise every sweep during a partial relay outage, so a new alarm email was not even attempted until the backlog parked.
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below. Email over the **SMTP** transport is further clamped per sweep to the active (lowest-Id) SMTP row's `MaxConcurrentConnections` when that value is positive, so a relay that limits connections is honoured; the EWS transport (which does not use the field) and SMS keep the full `MaxParallelDeliveries`. A sweep also has a **wall-clock budget** (`DispatchSweepBudget`, default 2 min): once it is spent, claimed notifications that have not yet started an attempt are skipped — left untouched and still due, so the next sweep re-claims them alongside newer rows — while attempts already in flight run to completion.
4. Applies the result:
Every timestamp written for an attempt (`LastAttemptAt`, `DeliveredAt`, `NextAttemptAt` = attempt instant + `RetryDelay`, and the attempt/terminal audit rows' occurrence time) is the **attempt instant**, captured after the adapter returns; the sweep-start instant is used only for the due-claim predicate. Stamping sweep-start skewed `DeliveredAt` for the tail of a slow sweep and scheduled `NextAttemptAt` in the past, collapsing the retry delay.
- **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
- **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
- **permanent failure** → `Parked`, record `LastError`.
@@ -45,13 +45,13 @@ The SMTP configuration is defined centrally and used by the central Email delive
- **Port**: SMTP port (e.g., 587 for StartTLS, 465 for SSL).
- **Authentication mode**: One of:
- **Basic Auth**: Username and password. For on-prem SMTP relays or servers that support basic authentication.
- **OAuth2 Client Credentials**: Tenant ID, Client ID, and Client Secret. For Microsoft 365 and other modern SMTP providers that require OAuth2. The Email adapter handles the token lifecycle internally (fetch, cache, refresh on expiry).
- **OAuth2 Client Credentials**: Tenant ID, Client ID, and Client Secret. For Microsoft 365 and other modern SMTP providers that require OAuth2. The Email adapter handles the token lifecycle internally (fetch, cache, refresh on expiry). The token cache (`OAuth2TokenService`, a singleton) is keyed per credential/authority/scope; an entry idle for more than an hour whose token has expired (or was never acquired) is evicted on the next token request, so a credential rotation does not leave the old credential cached for the life of the process.
- **OAuth2 authority** (optional): The token-endpoint URL requested during the client-credentials grant. When omitted it defaults to the Microsoft 365 endpoint `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token`, with the `{tenant}` placeholder substituted from the credential's tenant id; set it explicitly to target another identity provider (e.g. a non-M365 OAuth2 SMTP provider).
- **OAuth2 scope** (optional): The scope requested from the token endpoint. When omitted it defaults to the Microsoft 365 scope `https://outlook.office365.com/.default`; set it explicitly for another provider. Both fields are managed by Admin-role users via the CLI (`notification smtp update --oauth2-authority`/`--oauth2-scope`) and the Central UI `/notifications/smtp` page (the two inputs appear only for the OAuth2 auth type); a blank value is stored as null so the M365 default applies. Together they make "and other modern SMTP providers" true — a non-Microsoft OAuth2 relay is now configurable without a code change.
- **TLS mode**: None, StartTLS, or SSL.
- **From address**: The sender email address for all notifications (e.g., `scada-notifications@company.com`).
- **Connection timeout**: Maximum time to wait for SMTP connection (default: 30 seconds).
- **Max concurrent connections**: Maximum simultaneous SMTP connections from the central cluster (default: 5).
- **Max concurrent connections**: Maximum simultaneous SMTP connections from the central cluster (default: 5). Enforced by the Notification Outbox dispatcher, which clamps each sweep's Email parallelism to `min(MaxParallelDeliveries, MaxConcurrentConnections)` for the SMTP transport (a non-positive value applies no clamp).
- **Retry settings**: Max retry count, fixed time between retries. The Notification Outbox reuses these for delivery retry of transient failures.
- **Transport**: The submission mechanism — `Smtp` (default) or `Ews`. Null/empty means `Smtp`, so every pre-existing row keeps its behavior. See EWS Transport below.
+34 -8
View File
@@ -126,7 +126,7 @@ Violations from both passes are merged and deduplicated before being returned.
#### `Compile(string code, Type? globalsType = null, IEnumerable<MetadataReference>? extraReferences = null, IEnumerable<string>? extraImports = null)`
- Creates a `CSharpScript` with the given code, `globalsType`, references (defaults from `ScriptTrustPolicy.DefaultReferences` plus `extraReferences`), and imports (defaults from `ScriptTrustPolicy.DefaultImports` plus `extraImports`).
- Calls `.Compile()` and returns the resulting `Diagnostic[]` filtered to **error-severity diagnostics only**. This is a compile *gate*: a warning must not block a deploy, so only errors (undefined symbols, type mismatches) are surfaced to callers as gate failures.
- Reads `Script.GetCompilation().GetDiagnostics()` (no emit, no assembly load — see the verdict-cache notes under REQ-SA-4) and returns the diagnostics filtered to **error-severity diagnostics only**. Non-diagnostic faults are thrown, never returned as a message. This is a compile *gate*: a warning must not block a deploy, so only errors (undefined symbols, type mismatches) are surfaced to callers as gate failures.
- Each caller passes its own `globalsType` — `ScriptCompileSurface` for the design-time deploy gate, the real `ScriptGlobals` for Site Runtime execution, `null` for pure syntax checks.
#### `ParseDiagnostics(string code)`
@@ -134,6 +134,12 @@ Violations from both passes are merged and deduplicated before being returned.
- Parses the script text using Roslyn's `CSharpSyntaxTree.ParseText` and returns **error-severity** syntax-level diagnostics only (consistent with `Compile` — warnings do not fail the gate).
- No compilation is performed — useful for fast syntax checks where no globals type is available.
#### `BlockingWaitAnalyzer.Find(string code, Type? globalsType)` — advisory
Review 2026-10-21 01/SR-U2. Returns one **advisory** message per synchronous wait on a task in a script body — `.Result` / `.Wait(...)` on `Task`/`ValueTask` (and subclasses), `Task.WaitAll`/`WaitAny`, and `GetResult()` on a `System.Runtime.CompilerServices` awaiter (`.GetAwaiter().GetResult()`, including through `ConfigureAwait`). Site scripts run on the dedicated `ScriptExecutionScheduler` pool and not every runtime helper completes off that pool, so `var r = Instance.CallScript("Calc").Result;` in enough concurrent runs starves the pool until the stuck-script watchdog detaches workers. Resolution is semantic against the caller's `globalsType`, so a non-task member named `Result`/`Wait` is not flagged.
It is deliberately **not** a `ScriptTrustPolicy` violation: every trust-policy entry blocks deploy, and promoting this to an error would fail the next redeploy of any already-deployed script that blocks (none of the rig seed scripts do, but customer scripts may). The Template Engine surfaces it as a non-blocking `ScriptCompilation` **Warning** on validation (`ScriptCompiler.FindAdvisories`, memoised under its own cache surface key, a faulting analysis yields no advisory). Surfacing it as an editor marker in the Central UI is a Central UI concern.
---
### REQ-SA-4: Compile-Only Globals Stubs
@@ -167,13 +173,31 @@ generations of 2048 entries. New verdicts land in the *hot* generation; a hit in
*cold* is promoted back into hot; on overflow hot becomes the new cold and only the
old cold — the half nothing has touched for a full generation — is dropped.
The bound must never be enforced by clearing the cache outright. That reads as
harmless ("a verdict is cheap to recompute") but is not: a recompute is a fresh
Roslyn compile, and every script compile loads an assembly through a
**non-collectible `InteractiveAssemblyLoader`**. Bounding that leak is the entire
reason this cache exists, so dropping every hot entry at the 4096th distinct script
would re-open it for the whole working set at once. Segmenting retains anything in
active use across an eviction while keeping the same ceiling.
The bound must never be enforced by clearing the cache outright: a recompute is a
fresh trust analysis plus Roslyn compile against the full reference set (tens of ms
each), so dropping every hot entry at the 4096th distinct script would re-pay that
for the whole working set at once. Segmenting retains anything in active use across
an eviction while keeping the same ceiling.
**The gate never emits or loads an assembly (review 2026-10-21 04/M1).**
`RoslynScriptCompiler.Compile` reads `Script.GetCompilation().GetDiagnostics()` —
declaration and method-body errors, no PE emit — and passes every
`CSharpScript.Create` one shared `InteractiveAssemblyLoader` that never loads
anything. (It used to call `Script.Compile()`, which builds the executor: emit, then a
load through a fresh **non-collectible** `AssemblyLoadContext` per *distinct* script —
and `CSharpScript.Create` with no loader argument registers a new ALC even without the
load.) Pinned by a test that N distinct compiles leave `AssemblyLoadContext.All` and
the loaded-assembly count flat. The verdict cache therefore bounds CPU, not memory.
The site runtime's `ScriptCompilationService` is unaffected — it needs an executable
`Script` and keeps its own compile.
**Only diagnostic verdicts are cached (review 2026-10-21 04/C1).** `Compile` no longer
catches exceptions and returns them as a `"Compilation exception: …"` message; an
unexpected fault (metadata I/O, out-of-memory, …) propagates. The cache never stores a
verdict whose factory threw, and `ScriptCompiler.TryCompile` / `CheckExpressionSyntax`
turn the fault into an uncached failure for that call alone ("could not be checked —
the compile gate faulted … retry"), so a one-off fault can no longer become a sticky,
process-wide "failed to compile" on one central node until restart.
#### Parity guard
@@ -196,6 +220,8 @@ All five call sites that previously maintained their own script trust enforcemen
| **Central UI** `ScriptAnalysisService` | Semantic + full compile, lenient threading | Delegates forbidden-API verdict and sources editor-marker deny-list from `ScriptTrustPolicy`; retains Test-Run execution host |
| **Transport** `BundleImporter.RunSemanticValidationAsync` / `DetectBlockersAsync` | No trust gate — a bundle's forbidden-API scripts imported clean, deferred to runtime (fifth write path) | Runs `ScriptTrustValidator.FindViolations` over every non-Skip template / shared / ApiMethod body, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** at import review — hard error (apply) / Blocker row (preview) for all kinds |
**Trust verdicts are memoised (review 2026-10-21 04/P3).** `ScriptTrustValidator.FindViolations(code)` — the default-reference-set entry point every consumer above uses — caches its verdict process-wide by SHA-256 of the code (`TrustVerdictCache`: two generations of 1 024, cold-hit promotion, a throwing analysis is never stored; verdicts are returned as read-only collections and a hit returns the same instance). The verdict is a pure function of the code because the policy and `AnalysisReferences` are process-static. This removes the repeat full-framework semantic analysis the Transport importer used to pay on preview **and** again on apply (and on every wizard re-entry to preview) with no change at its call sites. A call that passes non-empty `extraReferences` (Central UI's run gate) bypasses the cache, since extra references can change resolution; the explicit-`baseReferences` overload is never cached.
The static enforcement is **defence-in-depth**, not a true runtime sandbox. The denied API list blocks disallowed APIs at compile time but does not establish an operating-system security boundary. Site-runtime consumers execute in-process. Central UI Test Run uses a bounded, killable worker process; that adds resource/lifetime containment, while the worker retains the same operating-system identity and permitted gateway access.
---
+22 -21
View File
@@ -34,25 +34,26 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
## Session Management
### Cookie + JWT Hybrid
- On successful authentication, the app issues a **JWT** signed with a shared symmetric key (HMAC-SHA256). Both central cluster nodes use the same signing key from configuration, so either node can issue and validate tokens.
- The JWT is embedded in an **authentication cookie** rather than being passed as a bearer token. This is the correct transport for Blazor Server, where persistent SignalR circuits do not carry Authorization headers — the browser automatically sends the cookie with every SignalR connection and HTTP request.
- The cookie is **HttpOnly** and **Secure** (requires HTTPS).
- On each request, the server extracts and validates the JWT from the cookie. All authorization decisions are made from the JWT claims without hitting the database.
- **JWT claims**: User display name, username, list of roles (`Administrator`, `Designer`, `Deployer`, `Viewer`), and for site-scoped Deployer, the list of permitted site IDs.
### Interactive sessions — cookie claims ticket
- On successful interactive login (`POST /auth/login`) the app signs the user in with ASP.NET Core **cookie authentication**. The cookie carries a data-protected **claims ticket** (no embedded JWT): display name, username, roles (`Administrator`, `Designer`, `Deployer`, `Viewer`, `Operator`, `Verifier`), the permitted site IDs of a site-scoped Deployer, the user's raw LDAP groups (`zb:group`) and the idle / role-refresh anchors. The claim shape is built by one shared `SessionClaimBuilder`.
- A cookie is the correct transport for Blazor Server: persistent SignalR circuits do not carry Authorization headers, and the browser sends the cookie with every SignalR connection and HTTP request.
- The cookie is **HttpOnly**, `SameSite=Strict`, and **Secure** (requires HTTPS) unless `RequireHttpsCookie` is explicitly turned off for a plain-HTTP dev rig.
- Both central nodes share the ASP.NET Data Protection key ring, so either node can read a cookie the other issued.
### Token Lifecycle
### CLI / API sessions — bearer JWT (`POST /auth/token`)
- `POST /auth/token` (form `username`/`password`) performs the LDAP bind (login-throttled), maps the groups to roles and site scope, and returns a short-lived **HS256 JWT** signed with `SecurityOptions.JwtSigningKey`, shared by both central nodes: `{ access_token, token_type: "Bearer", expires_in, management_bearer: true, username, display_name, roles }`.
- **JWT claims**: display name, username, roles, and for a site-scoped Deployer the permitted site IDs (no site claims = system-wide, the same normalisation the Basic path uses). Issuer and audience are both fixed to `scadabridge-central`. Lifetime is `JwtExpiryMinutes` (default **15**).
- `ManagementAuthenticator` accepts `Authorization: Bearer <jwt>` alongside HTTP Basic on every CLI-facing surface: `POST /management`, the audit REST endpoints, and the debug-stream hub (`/hubs/debug-stream`, token in the `Authorization` header of the negotiate request and WebSocket upgrade). Validation checks the HS256 signature, issuer, audience and lifetime (30 s clock skew for node-to-node drift; `nbf` is back-dated by the same skew so a peer whose clock lags the issuer accepts a fresh token after failover); unsigned (`alg=none`) or other-algorithm tokens are rejected, and a failure is a plain `401 AUTH_FAILED` (no LDAP bind, so it does not feed the login throttle). The token is never logged.
- **Roles and scope inside a valid token are authoritative for its lifetime**: a bearer request performs no LDAP bind and no role re-map. A role revocation therefore reaches a bearer caller when its token expires (≤ `JwtExpiryMinutes`); rotating `JwtSigningKey` invalidates every outstanding token immediately.
- The CLI exchanges credentials once per process, keeps the token in memory only, re-exchanges once on a `401`, and falls back to HTTP Basic when the server has no bearer support — see `Component-CLI.md`. HTTP Basic stays supported on every surface.
- Under the dev/test `DisableLogin` flag `/auth/token` answers `404`: the management surfaces already authenticate every request as the dev user, and no credential-free token is minted that another deployment sharing the key might honour.
- **Signing key**: `StartupValidator` refuses to boot a Central node whose `JwtSigningKey` is missing or shorter than **32 UTF-8 bytes** (256 bits, the HMAC-SHA256 floor). `JwtTokenService` repeats the length check at construction as defence in depth.
> **Implementation note (M2.19, #15).** The interactive Central UI login path signs in
> with **bare cookie claims**, not a cookie-embedded JWT. The session lifecycle below is
> therefore enforced by the cookie middleware (`ExpireTimeSpan` + `SlidingExpiration`) plus
> a `CookieAuthenticationEvents.OnValidatePrincipal` handler — see **Session Validation
> (`OnValidatePrincipal`)** below. The embedded-JWT model remains the documented design
> intent and is the mechanism for any non-cookie bearer surface (e.g. `/auth/token`), but
> it is **not** the transport for the cookie principal.
### Cookie session lifecycle
- **Idle timeout**: Configurable, default **30 minutes**. Without eligible activity within the idle window, the session is rejected and the user must re-login. Passive `/auth/ping`, activity prevalidation, and Blazor transport requests preserve the activity anchor and suppress sliding renewal; the activity endpoint renews only after authentication and antiforgery validation. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other.
- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads, and mutation entrypoints require a successful current mapping before dispatch.
- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads — display reads (`GetAuthenticationStateAsync`) at most once per `CookieAuthenticationStateProvider.NonAuthoritativeRefreshInterval` (10 s) per circuit — and mutation entrypoints (`GetAuthoritativeStateAsync`) require a successful current mapping before every dispatch.
- **Role-mapping cache**: `RoleMapper` reads the LDAP-group→role table and the scope rules of every Deployer mapping through a process-wide `RoleMappingCache` snapshot (`SecurityOptions.RoleMappingCacheSeconds`, default **30**; `0` disables). One refresh loads the whole table plus all Deployer scope rules, so a cache hit costs no database round-trip — previously every Basic-authenticated management request and every circuit auth-state read issued `GetAllMappingsAsync` plus one `GetScopeRulesForMappingAsync` per matched Deployer mapping. Without an explicit invalidation a mapping or scope-rule change is therefore visible on every node within the TTL. `RoleMappingCache.Invalidate()` discards the calling node's snapshot so its next mapping re-reads the database (a refresh started before an invalidation is never published over it); other nodes still rely on the TTL. `RoleMappingCacheSeconds = 0` bypasses the snapshot and reads only the matched mappings' scope rules per call.
#### Session Validation (`OnValidatePrincipal`)
- The cookie principal is built at login by a **single shared claim-builder** (`SessionClaimBuilder`). The `OnValidatePrincipal` role-refresh path rebuilds the principal through the **same** builder, so the login and refresh claim shapes cannot drift.
@@ -69,13 +70,13 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
> reflected during HTTP refresh and immediately at circuit mutation checks; directory group-membership changes require re-login.
### Load Balancer Compatibility
- The authentication cookie carries a self-contained JWT — no server-side session state. A load balancer in front of the central cluster can route requests to either node without sticky sessions or a shared session store.
- Since both central nodes share the same JWT signing key, either node can validate the cookie-embedded JWT. Central failover is transparent to users with valid cookies.
- Neither the cookie nor the bearer token needs server-side session state. A load balancer in front of the central cluster can route requests to either node without sticky sessions or a shared session store.
- Both central nodes share the Data Protection key ring (cookie) and the `JwtSigningKey` (bearer token), so central failover is transparent to a valid cookie or token.
## LDAP Connection Failure
- **New logins**: If the LDAP/AD server is unreachable, login attempts **fail**. Users cannot be authenticated without LDAP.
- **Active sessions**: Users with a valid (not-idle-timed-out) session can **continue operating** with their current roles during an LDAP outage. Interactive cookie sessions never re-query LDAP mid-session (the mid-session role-mapping refresh is DB-only — see *Session Validation* above), so a brief LDAP outage does not disrupt engineers mid-work; central role-mapping changes still apply within the refresh window regardless of LDAP availability.
- **Active sessions**: Users with a valid (not-idle-timed-out) session can **continue operating** with their current roles during an LDAP outage. Interactive cookie sessions never re-query LDAP mid-session (the mid-session role-mapping refresh is DB-only — see *Session Validation* above), so a brief LDAP outage does not disrupt engineers mid-work; central role-mapping changes still apply within the refresh window regardless of LDAP availability. A CLI holding a bearer token likewise keeps working until the token expires; HTTP Basic calls bind per request and fail during the outage.
- **Recovery (group-membership changes)**: Because the mid-session refresh is LDAP-free, a change to a user's **directory group membership** is picked up at the user's **next login** (when LDAP is queried again), not mid-session — see the *Residual limitation* note above.
## Roles
@@ -200,14 +201,14 @@ Role checks are expressed as named ASP.NET Core authorization policies (in `Auth
- **Deployment Manager**: Deployer role enforcement with site scoping.
- **Management Service / Central UI (secured writes)**: `RequireOperator` gates submission and `RequireVerifier` gates approval/rejection of MxGateway secured writes; the no-self-approval rule (Operator ≠ Verifier) is enforced server-side by the ManagementActor, not just in the UI.
- **All central components**: Role checks are a cross-cutting concern applied at the API layer.
- **Management Service**: The ManagementActor enforces role-based authorization on every incoming command using the authenticated user identity carried in the message envelope. The CLI authenticates users via the same LDAP bind mechanism and passes the user's identity (username, roles, permitted sites) in every request message. The ManagementActor applies the same role and site-scoping rules as the Central UI — no separate authentication path exists on the server side.
- **Management Service**: The ManagementActor enforces role-based authorization on every incoming command using the authenticated user identity carried in the message envelope. `ManagementAuthenticator` resolves that identity from either a bearer JWT (issued by `/auth/token` after the same LDAP bind + role mapping) or an HTTP Basic LDAP bind, and the identity (username, roles, permitted sites) travels in every request message. The ManagementActor applies the same role and site-scoping rules as the Central UI — no separate authentication path exists on the server side.
- **Transport (#24)**: Provides the `RequireDesign` policy (export) and `RequireAdmin` policy (import) enforced at both the Razor page layer and inside the `ZB.MOM.WW.ScadaBridge.Transport` service entrypoints.
## Login throttling
To blunt online password-spray and brute-force against the directory, every LDAP-bind surface consults a shared in-memory throttle (`LoginThrottle`, a `ZB.MOM.WW.ScadaBridge.Security` singleton) before attempting a bind, and records the bind outcome afterwards.
- **Scope — every LDAP-bind surface**: the Central UI interactive login (`POST /auth/login`) and CLI token issue (`POST /auth/token`), plus the HTTP-Basic management/CLI surfaces fronted by `ManagementAuthenticator` — `POST /management`, the audit REST endpoints, **and the debug-stream hub (`DebugStreamHub.OnConnectedAsync` delegates its whole credential path to `ManagementAuthenticator.AuthenticateAsync`, closing the round-2 N1 gap where the hub bound LDAP directly and bypassed the throttle)**. No bind happens outside these paths. The dev/test `DisableLogin` bypass sits **above** the throttle: a login-disabled deployment never binds, so it is never throttled.
- **Scope — every LDAP-bind surface**: the Central UI interactive login (`POST /auth/login`) and CLI token issue (`POST /auth/token`), plus the HTTP-Basic branch of the management/CLI surfaces fronted by `ManagementAuthenticator` — `POST /management`, the audit REST endpoints, **and the debug-stream hub (`DebugStreamHub.OnConnectedAsync` delegates its whole credential path to `ManagementAuthenticator.AuthenticateAsync`, closing the round-2 N1 gap where the hub bound LDAP directly and bypassed the throttle)**. No bind happens outside these paths; a bearer-token request binds nothing and is not throttled. The dev/test `DisableLogin` bypass sits **above** the throttle: a login-disabled deployment never binds, so it is never throttled.
- **Key and window**: counters are keyed per `{username}|{IP}` pair (lower-cased username; the IP half is the **forwarded-header-resolved client IP** when `ScadaBridge:Security:ForwardedHeaders` names a trusted proxy, otherwise the raw connection peer). A fixed window of `LoginFailureWindowMinutes` (default 5) opens on the first failure; reaching `MaxLoginFailuresPerWindow` failures (default 5) within it locks that key out for `LoginLockoutMinutes` (default 5). A successful bind clears the key; an expired window resets the count. Setting `MaxLoginFailuresPerWindow` to `0` disables throttling entirely.
- **Per-IP spray ceiling (arch-review 2026-08-19 P6-9).** The per-pair key alone is blind to a *spray* — one attacker IP trying one password against many distinct usernames never reaches `MaxLoginFailuresPerWindow` on any single pair — so a secondary counter keyed on the IP alone, same window/lockout but a higher ceiling (`PerIpMaxFailures`, default 25, validated strictly greater than `MaxLoginFailuresPerWindow`), locks out *new* attempts from that IP once it is reached; a successful bind does **not** reset the per-IP counter (only window/lockout expiry does), so a spray that stumbles onto one valid account cannot ride that success to keep spraying.
- **Proxy topologies and the username-lockout DoS.** The key is deliberately `{username}|{IP}`, not `{username}` alone: keying on username alone would let any network-adjacent actor lock any operator out of a SCADA control surface with five wrong passwords, repeatable indefinitely. That isolation only exists when the throttle sees real client IPs — behind a proxy **without** ForwardedHeaders trust, all clients collapse onto the proxy's IP and the DoS returns. Mitigations: (a) the shipped Traefik topologies enable trusted-proxy ForwardedHeaders (so per-IP isolation is real in the documented deployment); (b) the lockout window is short by default (`LoginLockoutMinutes` = 5) and never touches the directory account itself; (c) residual risk — an attacker spraying from their *own* IP locking out a victim username *at that IP only* — is the throttle working as designed. A success-path bypass ("a correct password unlocks") was considered and rejected: it would let an attacker who has the password bypass the lockout entirely, and it converts the throttle into a password oracle during the lockout window.
@@ -216,4 +217,4 @@ To blunt online password-spray and brute-force against the directory, every LDAP
### Passive traffic and circuit refresh (2026-09-09)
`/auth/ping`, `/auth/activity`, and Blazor transport validation do not advance inactivity timestamps or trigger sliding renewal. The explicit activity endpoint validates antiforgery before renewing the cookie and recording activity by protected session ID. A bounded process-local tracker lets a live circuit observe successful activity renewal. Role refresh preserves the session ID and activity timestamp; current circuit mutation checks re-map stored groups with fail-closed lookup behavior.
`/auth/ping`, `/auth/activity`, and Blazor transport validation do not advance inactivity timestamps or trigger sliding renewal. The explicit activity endpoint validates antiforgery before renewing the cookie and recording activity by protected session ID. A bounded process-local tracker lets a live circuit observe successful activity renewal. Role refresh preserves the session ID and activity timestamp; current circuit mutation checks re-map stored groups with fail-closed lookup behavior. Display-only auth-state reads reuse the circuit's last revalidation for up to 10 s (review 2026-10-21 07/P-3), so an idle timeout or a revoked grant can show on a read-only surface for at most that long; it never reaches a mutation.
+41 -1
View File
@@ -135,8 +135,41 @@ Pinned)` on the EventStream (transition-only, mirroring
health-observable condition rather than a silent log line, and the latch clears
with `Pinned=false` once a later tick makes progress.
**The pull serves only what central has not acknowledged.** The site's
`ReadChangedSinceAsync` skips every tracking row whose *current* version
(`UpdatedAtUtc`) central already accepted through the cached-telemetry push, so in
steady state a reconciliation pull carries only genuinely lost telemetry rather than
a second copy of every lifecycle step — and a central failover, which resets the
in-memory cursors to `DateTime.MinValue`, no longer replays the whole tracking
retention window of every site. When central acks an `IngestCachedTelemetry`
packet (it commits the packet's `SiteCalls` upsert with its audit row), the site's
`SiteAuditTelemetryActor` records `(TrackedOperationId, snapshot UpdatedAtUtc)` in
the **node-local** `OperationTrackingCentralAck` table — deliberately a separate,
unregistered table, never a column on the LocalDb-replicated `OperationTracking`
table (same posture as `mirrored_secrets`): only the node that pushed saw central
accept it. The filter matches the version exactly, so any later change (even one
stamped by a clock that stepped back) is served again; acks are monotonic per row
and pruned with the tracking rows they describe. **Failover:** a site node that
has just become active holds no ack rows, so its pull serves every row once
(bounded by central's cursor); central's idempotent monotonic upsert absorbs the
duplicates and the ledger rebuilds from that node's own pushes. A lost ack only
costs one extra pull of that row.
**Each pulled page is one batched upsert.** `ISiteCallAuditRepository.UpsertManyAsync`
runs the same two per-row statements as `UpsertAsync` (insert-if-absent, then the
ownership-checked monotonic update) for up to 128 rows per command — a 500-row page
is 4 round trips instead of 500. A chunk that faults (an insert race, or an
ownership violation) is re-applied row by row through `UpsertAsync`, which is inert
for rows already applied; the rest of the page is still applied and the first
row failure is rethrown, so the site's cursor stays put exactly as before.
**Removed sites are pruned.** Each tick drops the cursor and pinned latch of any
site the enumerator no longer returns (a pinned one publishes its `Pinned=false`
transition first), so the dictionaries do not grow with site churn; a site that
reappears restarts from `DateTime.MinValue`.
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
every site, up to a page ceiling of network pulls each, one upsert per row — so
every site, up to a page ceiling of network pulls each, one batched upsert per page — so
it runs as a background task with a `PipeTo`-delivered completion message and a
single-flight guard, not as an actor message handler. A handler occupies the
actor for its whole duration, so a post-outage catch-up used to park telemetry
@@ -209,6 +242,13 @@ mirroring the Notification Outbox KPI shape:
- Stuck count — `Pending`/`Retrying` older than a configurable threshold
(default 10 minutes); display-only, no escalation.
Each snapshot (global, per-site, per-node) is **one conditional-aggregation
query** over the rows that contribute to at least one metric (non-terminal,
Parked, or Delivered/Failed inside the interval), with the oldest-pending age a
`MIN` over a `CASE` in the same scan — the same shape as the Notification
Outbox repository. (It used to cost 7 / 6 / 6 sequential round trips, paid by
the 10 s dashboard poll and the per-minute KPI sampler.)
## Retention
Daily purge of terminal rows (`Delivered`, `Failed`, `Discarded`) after a
@@ -194,9 +194,9 @@ Each event entry contains:
- Time range
- Instance ID
- Severity
- **Keyword search**: Free-text search on message and source fields (SQLite LIKE query). Useful for finding events by script name, alarm name, or error message across all instances.
- **Keyword search**: Free-text search on message and source fields (SQLite LIKE query). Useful for finding events by script name, alarm name, or error message across all instances. The `LIKE` is a leading-wildcard full scan, so a keyword search is **bounded to a time window**: a query with no `From` (or a range wider than `ScadaBridge:SiteEventLog:KeywordSearchMaxWindow`, default 7 days) is clamped server-side to `(To ?? now) - window`, not rejected (review 2026-10-21 02/P1).
- Results are **paginated** with a configurable page size (default: 500 events). Each response includes a continuation token for fetching additional pages. This prevents broad queries from overwhelming the communication channel.
- The site processes the query locally against SQLite and returns matching results to central.
- The site processes the query locally against SQLite and returns matching results to central. The query runs off the actor thread (`Task.Run` + `PipeTo`) on its own short-lived read connection — WAL readers do not block the single writer, so a slow scan can no longer hold the logger's write lock and overflow the bounded write queue (review 2026-10-21 02/P1). The storage-cap trim tracks its row count locally (re-read from the database after the first batch and every 10th batch) and runs `PRAGMA incremental_vacuum` once per tick rather than per batch (02/P4).
## Dependencies
+25 -7
View File
@@ -104,6 +104,8 @@ flowchart TD
>
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
>
> **The sweep reads metadata, not configs (review 2026-10-21 01/SR-P1).** Each tick compares only name / enabled / revision, but it used to read every row's `config_json` (~70 KB worst case) — on a 500-instance site ~35 MB of large-object-heap strings a minute, forever, on a perfectly healthy site. The tick now reads the `GetAllDeployedConfigMetadataAsync` projection (no `config_json`), decides its repairs on the actor thread exactly as before, and only then fetches the full row — one single-row read per instance it is about to start or restart — off-thread. When those rows land every decision is **re-validated**: a fetch this node has since mutated past is dropped whole by the same store-write generation guard, a row that vanished, was disabled or now carries a different revision than the one claimed is skipped, and the deploy-pipeline / running-actor guards are re-checked; a repair that does not act gives back its once-per-revision claim so the next tick re-evaluates it. Ticks stay suppressed until the fetch lands. The boot load keeps the full read — it runs once and needs every blob.
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
>
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. One guard is **not** in that list because it needed adding later: deferring to an in-flight deploy pipeline says nothing about a snapshot that landed *after* the pipeline finished, which is the stale-snapshot race — a tick whose read raced this node's own deploy/enable/disable/delete is discarded whole by the store-write generation check (see **Stale-snapshot guard** below). A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
@@ -136,13 +138,15 @@ flowchart TD
>
> **Residuals still open**, recorded precisely in the decision memo: a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch. **Pipelined and parallel since review 2026-10-21 01/SR-P3:** the warm used to be one serial task per batch, and batch k+1's warm started only after batch k's actors existed *and* the stagger delay had elapsed, so recovery time was the sum of every distinct compile plus every delay, on one thread. Now (a) when a batch is created the **next** batch's warm starts immediately, running during the stagger delay, and (b) a batch's distinct compile units (script bodies + Expression triggers, de-duplicated by code first so N instances of one template do not race N identical cold compiles) are compiled with `Parallel.ForEach` at up to half the cores (`DeployCompileValidator.WarmCompileCache`; `ScriptCompilationService`/`SiteScriptCompileCache` are thread-safe — `ConcurrentDictionary` with a double-checked eviction lock). Creation order and the stagger delay are unchanged — a batch is still created only once its timer has fired **and** its warm has landed — so the reconnection-storm protection is exactly as before. The warm also returns each config parsed, which creation reuses (01/SR-P5).
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
> **The compile cache is owned by the `ScriptCompilationService` singleton, not static (register row 100)**: `SiteScriptCompileCache` is an ordinary instance held by `ScriptCompilationService`, and "one Roslyn compile per script body per process" (N4) is exactly the statement that `AddSiteRuntime` registers that service as a **DI singleton** — the deploy gate (via `DeployCompileValidator`), every Instance Actor's `PreStart`, and `SharedScriptLibrary` all reach the one cache through the one service the Deployment Manager threads to them. The singleton lifetime is therefore load-bearing rather than incidental, and is pinned by a registration test (`SiteRuntimeServiceRegistrationTests`); downgrading it to Scoped/Transient would hand every consumer an empty cache and silently reintroduce WP3.1's recompile storms with no behavioural test failing. The class was previously `static`, which made its entry table and its `Hits`/`Count` counters one process-global blob every test class in the assembly wrote into; the `Clear()` method that existed only so tests could reset that blob is **removed** — it never had a production caller, and constructing a cache is the honest reset.
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to clear **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
>
> **Eviction is visible, the bound is configurable, and eviction never reclaims the assembly (review 2026-10-21 SR-M2)**: the bound is `ScadaBridge:SiteRuntime:ScriptCompileCacheMaxEntries` (default 1024, validated `>= 8`). Eviction discards only the cached `Script<object?>`; the assembly Roslyn loaded for it stays in the default `AssemblyLoadContext` for the process lifetime (Roslyn scripting offers no collectible loader), so a body that is evicted and later recompiled — every Instance Actor restart, sweep, enable or failover — loads a *second* permanent copy. A site whose distinct script + trigger-expression body count exceeds the bound therefore grows monotonically at the eviction × restart rate. Each sweep increments `SiteScriptCompileCache.Evictions` and logs one Warning (rate-limited to one per minute, carrying the count since the last report) naming the option to raise. Size it above the site's distinct-body count. **Faulted compiles are never cached**: only success and diagnostic failures (a verdict on the code) are memoised; an unexpected exception during compile returns a non-cacheable `Faulted` result so a transient fault is retried on the next call instead of sticking until eviction/restart.
> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
@@ -156,22 +160,28 @@ flowchart TD
> **Dead-child bookkeeping + two-signal deploy join (S6).** All Instance Actors are `Watch`ed; an unexpected `Terminated` evicts the actor ref from the singleton's map. **Deviation from the naive design (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is **NOT** reported on persistence alone — the config store can commit before the Instance Actor's asynchronous `PreStart` throws. The design therefore uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once **both** persistence has committed **and** that readiness signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic in-memory state back (the persisted-row rollback is deferred until the store commits so it cannot race the write).
> **A deploy's config is deserialized once (review 2026-10-21 01/SR-P5).** The compile warm, the synchronous gate, the DCL-connection ensure and the Instance Actor constructor each used to `JsonSerializer.Deserialize<FlattenedConfiguration>` the same ~70 KB JSON — four object graphs per instance, two of them on the singleton's own thread. The warm now parses it once off-thread (`DeployCompileValidator.TryParse`), and the parsed object rides `DeployCompileWarmed` → the gate (`Validate(FlattenedConfiguration, …)`) → the redeploy buffer → `CreateInstanceActor` → a new `InstanceActor` constructor overload that takes the parsed configuration (read-only, so sharing it is safe; the Props no longer retain the JSON string either). A config that does not parse is still the gate's readable `Configuration JSON failed to deserialize` rejection. Paths that start from a stored row (enable, self-heal, boot) parse once inside `CreateInstanceActor`; a stored row that does not parse is handed to the actor as JSON, so it still fails init through the S6 path exactly as before.
> **Runtime names are validated before anything is applied (review 2026-10-21 01/SR-C1).** The instance unique name becomes the Instance Actor's name, and every script / alarm / native-alarm-source canonical name becomes a child actor name (`script-…`, `alarm-…`, `native-alarm-…`). Akka refuses any of these that is not a legal path element (a space, `#`, `/`, `[`, `]`, a bare `%`, a non-ASCII letter, a leading `$`), and the refusal is a **synchronous** `InvalidActorNameException` from `Context.ActorOf` — on the Deployment Manager singleton's own thread for an instance name, where the default Restart directive used to stop **every** Instance Actor on the site, lose all in-flight deploy bookkeeping, and leave the deployer's Ask to time out (so a retry repeated the outage). The site compile gate (`DeployCompileValidator`) now checks all of those names with Akka's own `ActorPath.IsValidPathElement` and rejects the deploy with `Site deploy validation failed: Invalid runtime name: …` before any reset, persistence or actor work. It deliberately does **not** decide with the Commons mirror `ActorNameRules`: the mirror is currently stricter than Akka (it refuses `"`, `(` and `)`), so gating on it would reject names that deploy today. Behind the gate, `CreateInstanceActor` catches the exception and returns a failure that each caller reports — a Failed deploy, a failed enable, an Error site event for a self-heal start, a skipped instance in the boot batch (Error site event; the name is dropped from the deployed count), and for an enable the already-committed `is_enabled=1` is reverted — so a row stored before names were checked can never escalate to the singleton. **Data connection names** become `DataConnectionActor` names through the DCL's shared sanitiser `ConnectionActorNames.Sanitize` (ASCII letters/digits plus `-_.*+:@&=,!~';()`, everything else → `-`, so the result is always Akka-legal). Because that mapping is not injective (`PLC A` and `PLC-A` both become `PLC-A`), the gate also rejects a config whose connection names collide after sanitising, and the DCL manager fails just the later connection (Error log + `connection` site event) instead of letting `ActorOf` restart it — the guard that also covers collisions between connections from different instances, which a per-instance gate cannot see. The Instance Actor applies the same rule to its children: a refused child name is logged and that one child skipped, exactly like a compile failure, instead of failing the whole instance's `PreStart`.
### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
- Recompiles shared scripts and makes updated code available to all Script Actors.
- **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.
- **Applies are serialised per site, in arrival order (review 2026-10-21 01/SR-C2).** Because each apply is "upsert my complete set, then delete every stored row not in it" and runs off the actor thread, two overlapping `DeployArtifactsCommand`s (central's per-site retry racing a fresh deploy, two operators clicking *Deploy Artifacts*) used to run concurrently, and the interleaving *B upserts Z → A deletes-except(A's set)* deleted Z — a shared script, external system, data connection or deployed secret central never removed — until the next artifact deploy. The Deployment Manager now keeps one apply in flight and queues the rest; the slot is released by a self-message sent after the apply task completes, i.e. behind that apply's own DCL push, so the final state is always exactly the last-arriving command's set. No revision comparison is made: a genuinely *older* snapshot that arrives last still wins, which is central's ordering to guarantee. An apply still holding the slot after 10 minutes is reported (Error log + `deployment` site event naming its deployment id and how many deploys are queued behind it) but **not** released, since releasing would let the next apply overlap a still-running one; and when the singleton stops, every still-queued deploy is answered `Success=false` rather than left to time out.
### Deployed Secrets (2026-09-28)
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. A current central always ships a list — **empty** when the site references no secrets — so dropping a site's last `secret:` reference removes the secret from both nodes. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
- The reply also carries `SecretsRemoved`: the names `UpsertDeployedSecretsAsync` soft-removed on this deploy (site-wide, whichever node's mirror later deletes them), or `null` when none. Central uses it for the `removed` list of its `DeploySiteSecrets` audit row, so a deploy that only retires secrets is audited centrally.
- **Ordering against the DCL.** The shipped secrets are written and mirrored **before** the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with **unchanged** connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
### Instance Lifecycle Commands
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup). An enable that arrives while a deploy/redeploy/delete pipeline still **owns** the instance past its compile warm (predecessor still terminating, or awaiting its persistence/init join) is refused with `A deployment for instance … is in progress; retry the enable after it completes.` — it used to race the deploy into `CreateInstanceActor`, collide with the terminating predecessor's actor name, and then revert `is_enabled`, leaving the redeployed instance durably disabled. A failed enable reverts `is_enabled` only for a name Akka refuses outright, never for a uniqueness collision on a valid name, which is transient (review 2026-10-21 follow-up L3).
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.
- **Ensure Started** (`EnsureInstanceStartedCommand`, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under **Startup Behavior**. Unlike the three commands above it has **no reply** and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
- **Instance reconcile sweep** (internal timer, not a command): the periodic singleton-owned check that the running Instance Actors still agree with the durable config rows — starting an enabled instance that has none, and re-applying a stored config a running instance has fallen behind. See the note under **Startup Behavior**; the start half shares its implementation (and therefore its bookkeeping and site events) with **Ensure Started**, and the restart half goes through the ordinary redeploy path.
@@ -223,10 +233,16 @@ flowchart TD
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context.
- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context. The same applies to the remaining script helper awaits (`CallScript`, `Alarms.CurrentAsync`, `ExternalSystem.Call`/`CachedCall`, `Database.Connection`, `Notify.Send`/`Status`, shared-script execution): their continuations run off the script scheduler, so a script that blocks on one with `.Result`/`.Wait()` cannot deadlock the pool against its own callee's completion.
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.
> **Routed multi-attribute reads and writes are one message each (review 2026-10-21 01/SR-P4).** `Route.To().GetAttributes()` / `SetAttributes()` used to cost one Ask — a temp actor, a mailbox round trip and a 30 s timer — per attribute. The Deployment Manager now sends the Instance Actor a single site-local `InstanceGetAttributesRequest` / `InstanceSetAttributesRequest` (`SiteRuntime/Messages/InstanceAttributeBatch.cs`, never on the wire). Reads are answered in one pass from one consistent point in the mailbox. Writes are applied in order exactly as the equivalent single `SetStaticAttributeCommand`s (static → in-memory + coalesced persist, data-sourced → DCL write, unknown/poison values rejected); data-sourced writes are issued concurrently and the reply waits only for those, each still bounded by its own 30 s DCL Ask (the batch Ask allows that 30 s plus 10 s of mailbox slack, so a device timeout or a busy instance mailbox surfaces as that attribute's own failure, not as a whole-batch timeout over writes that were applied). Both replies carry the per-attribute `GetAttributeResponse` / `SetStaticAttributeResponse`, so the routed response is unchanged: a failed device write or unknown attribute is still reported by name.
> **Loaded overrides are republished to the children and the stream (review 2026-10-21 01/SR-U1).** The override load is asynchronous and lands *after* `PreStart` has already created the Script/Alarm Actors, whose trigger snapshots are seeded from the attribute set at that moment — the deploy **defaults**. Static attributes never re-emit a change on their own, so before this fix every trigger on an instance that restarted, failed over, was enabled or was started by the reconcile sweep evaluated an overridden attribute at its default indefinitely (an Expression alarm on `Mode == "Auto"` never raised), and the Debug View live stream showed the default, while `GetAttribute` answered correctly. The load now republishes each override that changed the in-memory value or quality (quality from the load, no `SourceExecutionId`), with waiter evaluation **off** — nothing can legitimately be waiting on a replay. The site stream / Debug View receive it as an ordinary `AttributeValueChanged`; the children receive it as a site-local **`AttributeValueSeed`** (decision 2026-10-22): **Script Actors adopt the value without running** — the snapshot, a Conditional trigger's truth value and an Expression trigger's last result (evaluated, never fired) are brought up to date, so a restart/failover never re-fires ValueChange/Conditional/Expression scripts and the next real change is judged against the real value; a WhileTrue trigger whose condition already holds is seeded true without starting its re-fire timer. A seed evaluation whose task **faults** adopts nothing — it is not replayed as a real `false`, which would stop a running WhileTrue timer and make the next true evaluation fire as a spurious fresh edge (review 2026-10-21 follow-up L4). **Alarm Actors evaluate the seed like any change**, so an alarm whose condition already holds raises. A static write that lands before the load is newer than the stored row, so the load skips that attribute rather than rolling it back. Overrides for attributes the deployed config no longer defines are still applied (unchanged) but not republished.
> **Stopping an instance answers its pending waits (review 2026-10-21 01/SR-M3).** `PostStop` cancels every registered `WaitForAttribute` timeout and replies `Matched=false, TimedOut=false, ErrorMessage="… abandoned: instance stopped"`, so a script parked in `Attributes.WaitAsync` across a redeploy/disable/delete learns why immediately instead of burning its whole wait plus the Ask timeout and reporting a time-out, and no `WaitForAttributeTimeout` lands in dead letters (a monitored health metric).
### Debug View Support
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
@@ -283,7 +299,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **and a node-log Warning, both rate-limited to one per script per minute** (the log line carries the number of sheds since the previous one, so a 100 ms interval trigger against a wedged body cannot write 10 lines/s) so a hot trigger cannot flood `site_events` or the log, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
#### Traced runs (Flow Debugger, 2026-09-05)
@@ -426,16 +442,17 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
- **Display metadata persistence (UA4)**: persisted rows include the condition's display metadata, so **rehydrated conditions render fully** (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
- **Ack timestamp persistence**: the condition's `AckTime` rides the same `metadata_json` blob, so an acknowledged alarm keeps its ack instant across restart/failover rather than reappearing as if it had never been acknowledged. It is deliberately **not** a new physical column: `native_alarm_state` is `RegisterReplicated` in `SiteLocalDbSetup`, and LocalDb builds that table's CDC triggers from the column list at registration time — an additive JSON property changes no schema, no triggers and no replication contract, and `metadata_json` is exactly the extension point UA4 introduced for this. A row written before the field existed deserializes it as `null` ("ack time unknown"), which is the correct fallback. See `docs/plans/2026-06-30-mes-alarm-status-api.md` §6.4.
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; deletes ride the same dirty set (an upsert/delete discriminator per reference) and are applied in that one batched transaction — a snapshot swap dropping N stale conditions is one commit, not N — and a delete replaces a pending upsert for the same reference (no resurrection), as a later upsert replaces a pending delete. Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.
---
## Deployed Secrets (Site SQLite / LocalDb)
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Three tables, two different replication postures:
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
- **`secret_mirror_status`** — the **12th LocalDb-replicated table** (review finding M1). One row per site node: `node_name` (PK — `INodeIdentityProvider.NodeName`, i.e. `node-a`/`node-b`, falling back to the machine name when unconfigured), `failures_json` (a JSON array of `{SecretName, Message}` — the node's *current* mirror failures; names and messages only, never values or ciphertext), `checked_at`. No FKs. It is per-node *data* but replicated on purpose: each node writes only its own row, and replication is what lets the **active** node read the **standby's** row onto the site health report (only the active node sends one — see *Mirror status on the health report* below).
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
@@ -449,6 +466,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row, plus at least every `SiteRuntime:SecretMirrorStatusRefreshInterval` (default **10 min**) as a heartbeat — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have. **Stale rows are pruned:** the row is keyed by node name, so a decommissioned or renamed node used to leave its row (and any *Secret mirror failing* badge) behind forever. The reporter now leaves off the report, and deletes, any row whose `checked_at` is older than `SiteRuntime:SecretMirrorStatusStaleAfter` (default **1 h**, validated `≥ 2 ×` the refresh interval) — the delete is conditional on the `checked_at` it read, so a node that re-wrote its row in the meantime keeps it, and it replicates like any other write. A node down longer than that also drops off the list (its absence already shows in the cluster member count) and writes its row again on its first pass after it returns. Two guards keep a live node's row from being lost to the prune: the reporter prunes nothing during its first refresh interval after start-up (on a pair restarted together — an upgrade from a build without the heartbeat — every row is still old until each node's first write has replicated), and each mirror pass checks with a local read that its own row still holds what it last wrote, re-writing it otherwise (a peer's prune racing this node's write can replicate a delete over it; found on the rig).
---
@@ -589,7 +607,7 @@ Scripts execute **in-process** with constrained access. The following restrictio
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms, Log), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Its slot is then handed back and reused by the next worker start, so a script that wedges on every run does not grow the scheduler's slot table (or start a fresh slot) for the process lifetime. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
+12 -8
View File
@@ -19,8 +19,8 @@ Site clusters only. The central cluster does not buffer messages.
- Park messages that exhaust their retry limit (dead-letter).
- Persist buffered messages to local SQLite for durability.
- Emit cached-call lifecycle telemetry to the central Site Call Audit component via the `ICachedCallLifecycleObserver` hook (one notification per attempt outcome) so the audit pipeline can record each status transition.
- Replicate buffered messages to the standby node via application-level replication over Akka.NET remoting.
- On failover, the standby node takes over delivery from its replicated copy.
- Buffered messages are replicated to the standby node as rows of the LocalDb-replicated `sf_messages` table (CDC over the LocalDb gRPC sync stream).
- On failover, the standby node takes over delivery from its replicated copy of `sf_messages`.
- Respond to remote queries from central for parked message management (list, retry, discard), including central-driven Retry/Discard of parked cached calls.
## Message Lifecycle
@@ -34,7 +34,7 @@ flowchart TD
C -->|Failure| E[Buffer message]
E --> F[Retry loop<br/>per retry policy]
F --> G{Retry outcome}
G -->|Success| H([Remove from buffer<br/>+ notify standby])
G -->|Success| H([Remove from buffer<br/>+ CDC to standby])
G -->|Max retries exhausted| I([Park message<br/>dead-letter])
classDef ok fill:#d5e8d4,stroke:#82b366,color:#111111;
@@ -89,9 +89,9 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
## Persistence
- Buffered messages are persisted to a **local SQLite database** on each site node.
- The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- The active node persists locally; every buffer mutation (add, remove, park) is a row change on `sf_messages`, which LocalDb CDC captures and ships to the standby **asynchronously**. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
- The standby node receives those row changes into its own copy of the table but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby's copy is kept warm purely for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent changes may not have replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its status change not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
@@ -123,9 +123,13 @@ On every tracking-table status transition, the site emits a `CachedCallTelemetry
The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel<T>` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry.
### Queue-depth gauge
`scadabridge.store_and_forward.queue.depth` reads a cached counter adjusted by this node's own enqueue/deliver/park paths. Rows that arrive via LocalDb CDC (standby) or change on the peer never touch it, so every 6th retry sweep (including the first) re-seeds it from the database's Pending count; the reported value is clamped at 0 (review 2026-10-21 02/C3).
### Retry Sweep Indexing
`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. The sweep page is ranked **per `(category, target)` lane** (`ROW_NUMBER() OVER (PARTITION BY category, target ORDER BY created_at)`, ordered by lane rank then `created_at`, limited to `SweepBatchLimit`), so a dead target with hundreds of due rows contributes one row per round like every other lane instead of owning the whole page and starving healthy targets (review 2026-10-21 02/C2); per-target FIFO is preserved. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
## Parked Message Management
@@ -153,7 +157,7 @@ Each buffered message stores:
## Dependencies
- **SQLite**: Local persistence on each node.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks.
- **Communication Layer**: remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks. (Standby replication is not carried by Communication — it is LocalDb CDC.)
- **External System Gateway**: Delivers external system API calls.
- **Central–Site Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over Central–Site Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.
+58 -8
View File
@@ -93,6 +93,23 @@ When a template composes a feature module, members from that module are addresse
- The composing template's own members (not from a module) have no prefix — they are top-level names.
- Naming collision detection operates on canonical names, so two modules can define the same member name as long as their module instance names differ.
### Runtime actor-name rules (review 2026-10-21 01/SR-C1)
Some names become Akka actor names at the site: the instance **unique name** (the
Instance Actor), and every **script**, **alarm** and **native alarm source**
canonical name (`script-…`, `alarm-…`, `native-alarm-…` children of the Instance
Actor) — which also embeds every **composition slot** name on its path. Akka accepts
only ASCII letters, digits and `-_.*$+:@&=,!~';` (plus `%HH` escapes) and no leading
`$`; anything else (a space, `#`, `/`, non-ASCII letters) makes `ActorOf` throw at
deploy — for an instance name, inside the site's Deployment Manager singleton. The
Template Engine therefore rejects such names **at save** with a readable error, using
the shared `Commons.Validation.ActorNameRules`: `InstanceService.CreateInstanceAsync`
(unique name), `TemplateService.AddScriptAsync` / `AddAlarmAsync` (member names are
immutable after creation, so updates need no check) and `AddCompositionAsync` /
`RenameCompositionAsync` (slot names). Attribute names and template names are not
actor names and are unrestricted. Native alarm sources are created outside
`TemplateService` (Management Service / Transport import); those paths must apply the same rule.
### Derived template naming
A composition slot is materialized as its own *derived* template. A derived
@@ -126,6 +143,8 @@ If any composed module introduces a name that already exists (from another compo
A derived template stores `IsInherited` placeholder rows that mirror the members it inherits from its parent chain. These placeholders are **excluded** from collision detection on the template's own member set and on the inherited-parent walk — the inheritance walk already re-adds those members under the parent's origin, so counting the placeholder copies as well would report a spurious self-collision against the very member they mirror. (Placeholders are only counted when a composed module is itself a derived template, because the composed-module walk does not climb that module's parent chain and the placeholders are then the sole representation of its inherited members.) Without this exclusion there is no supported way to add an attribute, alarm, script, or composition to a derived template — every add is blocked by false collisions on the pre-existing inherited rows.
A template composed into several slots — or reached through several parents (a diamond) — is checked under **every** slot, since each slot yields distinct canonical names. Its module-relative member list is computed once per collision check and memoised by template id; each further slot only re-prefixes it (review 2026-10-21 04/P8(b)). A genuine composition cycle still contributes nothing at the re-entered module, and a list cut short by a cycle is never memoised.
## Flattening Process
When an instance is deployed, the Template Engine resolves the full configuration:
@@ -182,11 +201,12 @@ not replicated between central nodes. A restart, a failover, or any mutation sim
misses and falls back to the authoritative full flatten, so the watermark can only
ever cause EXTRA work, never stale work.
The same watermark backs a **staleness fast path** in `StaleInstanceProbe`: a
previously computed revision hash is reused when the instance's version, the
structure version, and the version of every template that flatten walked are all
unchanged. This matters most where the probe is called per-instance across a whole
bundle import or a fleet-wide staleness sweep.
The watermark does **not** back any staleness memo. `StaleInstanceProbe` once reused
a previously computed revision hash while the watermark versions were unchanged; that
fast path was removed (MG-3), because the watermark is process-local and a
configuration edit made through the *other* central node never bumps it. Every probe
now starts a fresh flatten session — see `Component-DeploymentManager.md` →
*Authoritative staleness and bulk deadlines* (review 2026-10-21 04/U2).
### Design-time analysis reads
@@ -194,7 +214,15 @@ The design-time checks — acyclicity (`CycleDetector`), naming collisions
(`CollisionDetector`) and canonical-name resolution (`TemplateResolver`) — read the
whole template graph but write through none of it and read no script bodies. They
use `GetAllTemplatesForAnalysisAsync`: `AsNoTracking`, with `TemplateScript.Code`
projected away.
projected away. The same projection serves the non-analysis lookups that need only
names, locks and edges: `InstanceService.SetAlarmOverrideAsync` (resolving one alarm
name) and `TemplateDeletionService.CanDeleteTemplateAsync` (inheritor / derived /
composer checks). `TemplateFolderService.DeleteFolderAsync`, which only counts
templates per `FolderId`, reads the lighter `GetTemplateStubsAsync` identity projection
(review 2026-10-21 04/P6, 08/P1). Template member add/update paths still load the whole
analysis projection for collision detection; narrowing that to the edited template's
chain + composition closure needs a closure-scoped repository read that does not exist
yet.
The tracked, body-bearing `GetAllTemplatesAsync` remains the read for the two paths
that genuinely need it — the inheritance reconciler (which compares and copies
@@ -259,7 +287,7 @@ Before a deployment is sent to a site, the Template Engine performs comprehensiv
- **Flattening**: The full template hierarchy resolves and flattens without errors.
- **Naming collision detection**: No duplicate attribute, alarm, or script names in the flattened configuration.
- **Script compilation and trust check**: All instance scripts and alarm on-trigger scripts are compiled via `ScriptCompiler.TryCompile`, which performs a real Roslyn type-checking compile (using `ScriptCompileSurface` from the Script Analysis component as the globals type) and an authoritative forbidden-API check (via `ScriptTrustValidator.FindViolations`). Scripts with compilation errors, type errors, or forbidden-API violations are rejected. This gate is authoritative — not advisory — meaning alias, `using static`, and `global::` bypasses are caught by semantic symbol resolution.
- **Script compilation and trust check**: All instance scripts and alarm on-trigger scripts are compiled via `ScriptCompiler.TryCompile`, which performs a real Roslyn type-checking compile (using `ScriptCompileSurface` from the Script Analysis component as the globals type) and an authoritative forbidden-API check (via `ScriptTrustValidator.FindViolations`). Scripts with compilation errors, type errors, or forbidden-API violations are rejected. This gate is authoritative — not advisory — meaning alias, `using static`, and `global::` bypasses are caught by semantic symbol resolution. A script that compiles is additionally run through `BlockingWaitAnalyzer` (via `ScriptCompiler.FindAdvisories`): `.Result` / `.Wait()` / `.GetAwaiter().GetResult()` on a task produces a non-blocking `ScriptCompilation` **Warning** naming the script and line — it never gates deploy (review 2026-10-21 01/SR-U2).
- **Alarm trigger references**: Alarm trigger definitions reference attributes that exist in the flattened configuration.
- **Script trigger references**: Script triggers (value change, conditional) reference attributes that exist in the flattened configuration.
- **Expression-trigger syntax/trust check**: Each Expression-triggered script/alarm has its boolean expression run through `ValidationService.CheckExpressionSyntax` — a `ScriptTrustValidator.FindViolations` forbidden-API verdict followed by a real Roslyn compile against `TriggerCompileSurface`. Both stages report **all** findings (every forbidden API, every compile error), not just the first, so an operator fixing a multi-error expression sees the full list in one round-trip. The verdict is memoised in the surface-keyed `ScriptCompileVerdictCache`, so an unchanged expression compiles once per process.
@@ -302,7 +330,29 @@ The same validation logic is available to Design users in the Central UI without
### Shared Script Validation
For shared scripts, pre-compilation validation is performed before deployment. Since shared scripts have no instance context, validation uses `RoslynScriptCompiler.ParseDiagnostics` for syntax errors and `ScriptTrustValidator.FindViolations` for the forbidden-API check. Full type-checking compilation is not performed (no globals type is available), but syntax and trust violations are caught and reported.
`SharedScriptService` validates on **create and update** (review 2026-10-21 04/U3 — the MG-8 remediation made this a full gate): a cheap structural pass (`CSharpDelimiterScanner` — balanced delimiters outside strings/comments) followed by the **same authoritative gate as instance scripts**, `ScriptCompiler.TryCompile` — `ScriptTrustValidator.FindViolations` for the forbidden-API check plus a real Roslyn type-checking compile against **`ScriptCompileSurface`**, the instance-script globals surface. A save that fails either stage is rejected with the error text.
Gating against the instance surface is a design decision worth knowing: a shared script executes inline inside the calling script's context, so it is type-checked as if it were an instance script. A shared script that references an identifier the instance surface does not expose fails at save, and one that uses an instance accessor (`Attributes`, `Instance`, …) compiles — whether that accessor is meaningful at run time depends on the caller.
## Configuration Audit Shape
`IAuditService.LogAsync` serialises the after-state it is handed into the
indefinitely-retained `AuditLogEntries.AfterStateJson`. Template Engine services
therefore hand it a **projected summary** for aggregates, never a tracked entity
graph (review 2026-10-21 08/G1, `TemplateAuditShapes`):
- **Template** create / update / move — identity and scalar fields (`Id`, `Name`,
`Description`, `ParentTemplateId`, `FolderId`, `IsDerived`, `OwnerCompositionId`),
per-collection member counts, and `ChangedFields` (e.g. `Name`, `Description`,
`FolderId`). No member rows. A rename of a 40-script template used to write all 40
script bodies.
- **Template script** and **shared script** create / update — every scalar field
except the body, which is recorded as `CodeLength` + `CodeSha256`, so an auditor can
tell whether (and to what) a body changed without the table storing every revision
of every body.
Small leaf entities (attributes, alarms, compositions, folders, areas, instance
overrides) are still audited as-is.
## Dependencies
+11 -4
View File
@@ -134,9 +134,9 @@ flowchart TD
class T muted
```
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory. **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing. **A log line is not alertable, so register row 66 adds the counter `scadabridge.central.transport.import_sessions.rejected`** on the same meter, incremented at the same single genuine-rejection point (the post-eviction recheck — never on the reclaim-then-succeed path, and never when re-opening an already-open session id). Alert on the counter; interpret it against the gauge — rejections rising while the gauge stays below the cap means genuine concurrency (raise the cap), whereas rejections rising while the gauge is pinned at the cap means a leaked session (raising the cap only defers it).
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins its decrypted content in memory. **Resident footprint (review 2026-10-21 04/M2):** the session holds a Brotli-compressed copy of the decrypted content JSON — not the plaintext, and not a parsed DTO (the `SecretsBlock` values are plaintext once the content is decrypted, and immutable strings cannot be zeroed; a byte buffer can) — and `LoadAsync` zeroes the plaintext as soon as it is compressed; `PreviewAsync`/`ApplyAsync` deserialise straight from the compressed stream. The buffered upload zip is released once the manifest and content entry are extracted. Worst case per session is still bounded only by `MaxBundleEntryDecompressedMb` (200 MB — an adversarial incompressible payload), so the hard ceiling is `MaxConcurrentImportSessions` × ~200 MB (1.6 GB at the default 8); a real export compresses ~5–10x, so typically 8 × 20–40 MB at the largest bundle size. One in-flight `LoadAsync` additionally peaks at roughly the zip (≤ `MaxBundleSizeMb`) plus the content entry and its decrypted copy (each ≤ `MaxBundleEntryDecompressedMb`). **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing. **A log line is not alertable, so register row 66 adds the counter `scadabridge.central.transport.import_sessions.rejected`** on the same meter, incremented at the same single genuine-rejection point (the post-eviction recheck — never on the reclaim-then-succeed path, and never when re-opening an already-open session id). Alert on the counter; interpret it against the gauge — rejections rising while the gauge stays below the cap means genuine concurrency (raise the cap), whereas rejections rising while the gauge is pinned at the cap means a leaked session (raising the cap only defers it).
**Session release is explicit, not TTL-only (register row 43).** `ApplyAsync` releases the session it applied (zero-then-remove, T-007), and `IBundleImporter.Abandon(sessionId)` does the same for a session that will never be applied. Every abandonment path calls it: the Central UI wizard on new upload / Back-to-Upload / `ResetSessionState` / lockout reset / `Dispose` (navigation away or circuit teardown), the CLI `transport preview` handler (a preview never applies), and the CLI `transport import` handler for the aborts that precede apply (blocker rows, unresolved mappings). `Abandon` is idempotent — double release, an unknown id, and an already-expired id are all silent no-ops — and since session ids are random GUIDs handed only to their opener, abandoning the id you hold can never evict another operator's in-progress import. Relying on the TTL alone was an availability *and* secret-lifetime defect: eight abandoned wizards exhausted `MaxConcurrentImportSessions` and locked out every import on the active node for 30 minutes, each still holding its decrypted bundle content.
**Session release is explicit, not TTL-only (register row 43).** `ApplyAsync` releases the session it applied (zero-then-remove, T-007), and `IBundleImporter.Abandon(sessionId)` does the same for a session that will never be applied. Every abandonment path calls it: the Central UI wizard on new upload / Back-to-Upload / `ResetSessionState` / lockout reset / `Dispose` (navigation away or circuit teardown), the CLI `transport preview` handler (a preview never applies), and the CLI `transport import` handler for the aborts that precede apply (blocker rows, unresolved mappings). `Abandon` is idempotent — double release, an unknown id, and an already-expired id are all silent no-ops — and since session ids are random GUIDs handed only to their opener, abandoning the id you hold can never evict another operator's in-progress import. Relying on the TTL alone was an availability *and* secret-lifetime defect: eight abandoned wizards exhausted `MaxConcurrentImportSessions` and locked out every import on the active node for 30 minutes, each still holding its decrypted bundle content. **TTL expiry zeroes too (review 2026-10-21 04/C3):** a session dropped because it expired — lazily on `Get`, or by the `EvictExpired` sweep — has its content buffer zeroed by `BundleSessionStore` as it is removed, so every release path (apply, abandon, expiry) follows the same zero-then-remove discipline.
**Preview→apply window (recorded decision).** The diff/preview an operator reviews is a point-in-time snapshot; there is **no optimistic concurrency** between preview and apply. If the target config changes in the window between `PreviewAsync` and `ApplyAsync` (another admin edits a template, say), the apply proceeds against the current state — the resolutions are keyed by `(EntityType, Name)`, so they still target the right rows, but the operator may not have seen the latest state in the diff. This is accepted for v1 (single-operator import is the expected workflow, and the terminal Confirm step already gates a typo-resistant environment-name entry). The bundle-format `version` fields are reserved for a future optimistic-concurrency check but are not yet enforced (Underdeveloped 5).
@@ -159,6 +159,8 @@ A **Sites & Instances** section (T18) adds a flat list of sites, each row expand
The user can toggle "include all dependencies" off (with a warning that the bundle may produce an invalid import).
**Resolver round trips (review 2026-10-21 04/P5).** Instances are read with their override/binding collections in the same query (`GetInstanceByIdAsync` / the template repository's `GetInstancesBySiteIdAsync` both `Include` them — there is no per-collection reload); a site's data connections are read once per export and shared between the site seed and the per-instance closure; and missing templates (instance templates, then each level of the composition closure) are fetched in one batch through `GetTemplatesWithChildrenAsync` keyed by an id→name map from `GetTemplateStubsAsync` — a lone missing template still uses the by-id accessor. Exporting one 500-instance site is a few dozen queries rather than ~5 000.
**Step 3 — Encryption.** Passphrase and confirm. Strength meter. Explicit warning lists every secret field being encrypted. An "Export without encryption" path is logged with the `UnencryptedBundleExport` audit flag.
**Step 4 — Download.** Generated filename pattern: `scadabundle-{sourceEnv}-{yyyy-MM-dd-HHmmss}.scadabundle`. SHA-256 displayed for out-of-band verification.
@@ -299,6 +301,8 @@ Site identifiers and connection names are environment-specific, so a bundle expo
**Auto-match → operator override → apply.** The importer first auto-matches every source site and connection against the target by these identity keys. The operator confirms/overrides via the Central UI **Map step** (Step 3) or the CLI `--map-site` / `--map-connection` / `--create-missing-*` flags. References that neither auto-match nor are set to Create-new become **blocker rows** (UI) — or, on the CLI without `--create-missing-sites` / `--create-missing-connections`, abort before apply.
**Template library reads (review 2026-10-21 04/P4 + 08/P1).** A preview and each apply attempt read the target template library **once**, through body-free projections (`GetTemplateStubsAsync` for id/name maps, `GetAllTemplatesForAnalysisAsync` — untracked, members and lock flags but no script bodies — where members or edges are needed), held on a per-session `TemplateLibrary` and threaded to every consumer: preview diff names, blocker and locked-override scans, apply-time validation, the composition/inheritance rewire fallbacks (formerly a full library reload *inside* the per-template loop), the instance pass's name→id map (overlaid with the change tracker's just-flushed templates), the acyclicity guard and the stale-instance expansion. The full tracked graph is loaded only for the templates the bundle **Overwrites** (`GetTemplatesWithChildrenAsync` by name), since their child syncs mutate it. The acyclicity guard reads the change tracker's staged templates overlaid on that snapshot — the untouched part of the graph has not changed during the apply. Instance lookups are batched by name set (`ITemplateEngineRepository.GetInstancesByUniqueNamesAsync`, one tracked + hydrated read for preview and one for the instance pass, Rename targets included), and the preview resolves an existing instance's template / site / area names from the library, one site list and one area list per site, and reads shared scripts and external systems once each.
**Template second-pass rewires.** Bundles carry template-to-template references by **name**, not id (ids are environment-specific). After the templates are flushed, the importer runs three name-resolution passes over the imported templates so every cross-template FK points at the freshly-persisted target row: the **alarm on-trigger-script** links (`TemplateAlarm.OnTriggerScriptId`), the **composition** edges (`TemplateComposition`), and the **inheritance** edges (`Template.ParentTemplateId`, resolved from `BaseTemplateName` through the same rename/skip resolution map — a `null` base clears a stale edge on Overwrite). Each pass is best-effort: an unresolvable reference leaves the FK null and emits a warning audit row (`BundleImportAlarmScriptUnresolved` / `BundleImportCompositionUnresolved` / `BundleImportBaseTemplateUnresolved`) rather than aborting the import.
**Rename call-site limitation (load-bearing).** A `Rename` resolution renames the entity being imported (e.g. a SharedScript `Foo` → `Bar`), but it does **not** rewrite call sites inside importing scripts — a script body that calls `CallShared("Foo")` still says `Foo` after import. So the target DB can end up with scripts referencing the old name. This is deliberate: Pass 1 name resolution registers **both** the original and renamed names as resolvable (so the import isn't blocked), and the authoritative check is the target's own **deploy-time validation** (`ScriptCompiler.TryCompile` in the Template Engine deploy gate), which recompiles against the actual persisted catalog. Operators renaming a shared dependency must fix the calling scripts on the target (or re-export with the dependency included under its original name).
@@ -312,7 +316,7 @@ Site identifiers and connection names are environment-specific, so a bundle expo
There is no explicit stale-mark write. Overwriting a template during import changes its content, which naturally changes the flattened-config hash that `DeploymentService.CompareAsync` computes at query time. The Deployments page already surfaces any instance whose computed hash differs from the deployed snapshot — no new mechanism is required.
As of M8 the import result enumerates affected instances for real (closing the prior `StaleInstanceIds` stub). Before commit, for each **Overwritten** template the importer walks the existing **deployed** target instances of that template and, via the `IStaleInstanceProbe` seam in Commons (implemented in the Deployment Manager over the flattening pipeline), compares each instance's freshly-flattened revision hash against its `DeployedConfigSnapshot.RevisionHash`; instances whose hash has drifted are returned in `ImportResult.StaleInstanceIds`. The probe is optional — when it is not registered the result is an empty set (informational only, never blocking). Freshly-imported `NotDeployed` instances are surfaced as **new**, not stale. Step 5 (Confirm) shows the real count and Step 6 (Result) deep-links the Deployments page filtered to those instances.
As of M8 the import result enumerates affected instances for real (closing the prior `StaleInstanceIds` stub). Before commit the importer takes the set of **Overwritten** templates and — review 2026-10-21 04/C2 — expands it over the post-apply template graph to every template whose flattened config includes one of them: **derived templates** (via `ParentTemplateId`) and **templates that compose it** (via `TemplateComposition.ComposedTemplateId`), transitively. Overwriting a base or a feature module with no direct instances therefore still reports the deployed instances of everything built on it (previously only instances whose `TemplateId` was the overwritten id were probed, so such an import reported 0 stale instances and the operator was told nothing needed redeploying). It then walks the **deployed** target instances of every affected template and, via the `IStaleInstanceProbe` seam in Commons (implemented in the Deployment Manager over the flattening pipeline), compares each instance's freshly-flattened revision hash against its latest `DeployedConfigSnapshot.RevisionHash` (read for all candidates in one projected query); instances whose hash has drifted are returned in `ImportResult.StaleInstanceIds`. The probe is used through `IStaleInstanceProbe.BeginBatch()`, which shares one `FlattenSession` across the whole check so a template chain common to many instances is walked once. The probe is optional — when it is not registered the result is an empty set (informational only, never blocking). Freshly-imported `NotDeployed` instances are surfaced as **new**, not stale. Step 5 (Confirm) shows the real count and Step 6 (Result) deep-links the Deployments page filtered to those instances.
## Error Handling
@@ -326,7 +330,10 @@ As of M8 the import result enumerates affected instances for real (closing the p
| Map | `CreateNew` selected but the bundle does not carry that site/connection's config | Blocker — Create-new requires the full config to be present in the bundle payload |
| Preview | ApiMethod script references shared script / external system not in bundle and not in target DB | Listed as a **blocker** row in the Diff step; cannot Apply until resolved (no downstream deploy gate for ApiMethods) |
| Preview | Template script references shared script / external system not in bundle and not in target DB | Listed as an advisory **warning** row; import proceeds — the design-time deploy gate re-validates the script authoritatively |
| Preview | A bundled site's `GrpcNodeAAddress`/`GrpcNodeBAddress` is not empty and not an absolute `http(s)://host[:port]` URL (`GrpcEndpointRules`; review 2026-10-21 03/R-1) | **Blocker** row (`Site`) when the target has no such site (the import would create it); advisory **warning** when it does (only an Overwrite writes the addresses) |
| Preview | A name that becomes a site actor name fails `ActorNameRules` — an instance's unique name, or a template's script / alarm / composition-slot / native-alarm-source name | **Blocker** row per name (`Instance` / `Template` `{template}.{member}`), with the same readable message the Central UI save path gives |
| Apply | Semantic validation fails (call target type mismatch, etc.) | Modal: "Validation failed — N errors", per-error list, no DB writes |
| Apply | Either rule above is violated by an entity the apply would write — skip-resolved entities are exempt; an instance's Rename target is checked; a site's endpoints are checked when it is created or Overwritten | Same `SemanticValidationException` path, raised before any row is staged — never a mid-apply exception, a central channel-build throw, or an `ActorOf` throw at deploy |
| Apply | DB transaction fails | Rollback; full transactional guarantee — nothing partial lands; `BundleImportFailed` audit row written outside the rolled-back transaction |
| Session | TTL expired | Diff step onward, refresh prompts re-upload |
@@ -498,7 +505,7 @@ The `manifest.json` file is always present in the ZIP root and is never encrypte
| Field | Description |
|---|---|
| `bundleFormatVersion` | Integer. Importer hard-refuses any value higher than what its `TransportOptions.SchemaVersionMajor` supports. Stays `1` through T18. |
| `bundleFormatVersion` | Integer. Importer hard-refuses any value other than the code constant `ManifestBuilder.CurrentBundleFormatVersion`. Stays `1` through T18. The schema and format versions are code constants by design — there is no `Transport:SchemaVersionMajor` option (removed 2026-10-22; it was validated but never read). |
| `schemaVersion` | Semver string. Minor increments are additive-only and accepted by older importers. T18 bumped this `1.0 → 1.1` (adding the `sites`/`dataConnections`/`instances` content arrays); a pre-T18 importer lists those new entries as "skipped — unsupported in this version" since `ManifestValidator` gates only on `bundleFormatVersion`. |
| `createdAtUtc` | ISO-8601 UTC timestamp of when the export was created. |
| `sourceEnvironment` | The `SourceEnvironment` name of the exporting cluster (from `TransportOptions`). Displayed in the import wizard and required to be retyped at the confirm step. |
+1 -1
View File
@@ -437,7 +437,7 @@ Audit logging is implemented as part of the **Configuration Database** component
### 10.1 Storage
- Audit logs are stored in the **configuration MS SQL database** alongside system config data, enabling direct querying.
- Entries are **append-only** — never modified or deleted. No retention policy — retained indefinitely.
- Entries are **append-only** — never modified or deleted. Retained indefinitely by default; an optional `ScadaBridge:ConfigAudit:RetentionDays` (default `0` = off) enables a daily batched purge of older entries.
### 10.2 Scope
All system-modifying actions are logged, including:
+1 -2
View File
@@ -71,7 +71,6 @@ that is the same variable, not a separate one).
| Variable | In appsettings? | Config key | Purpose | Potential values | Required |
|---|---|---|---|---|---|
| `ScadaBridge__Database__ConfigurationDb` | Yes — `${secret:sql/scadabridge/configdb-connection}` in `appsettings.Central.json` | `ScadaBridge:Database:ConfigurationDb` | Central config DB connection string (MS SQL). Central-only. | `Server=host,1433;Database=ScadaBridgeConfig;User Id=...;Password=...;TrustServerCertificate=true` | Yes on Central — via env **or** the `${secret}` |
| `ScadaBridge__Database__MachineDataDb` | No in shipped src appsettings; present in the deploy overlay (`deploy/wonder-app-vd03/appsettings.Central.json`, as `${...}`). install.ps1 derives it as `<DbName>MachineData`. | `ScadaBridge:Database:MachineDataDb` | Central machine-data DB connection string. Central-only. | connection string | Yes on Central |
| `ScadaBridge__Security__JwtSigningKey` | Yes — `${secret:security/scadabridge/jwt-signing-key}` in `appsettings.Central.json` | `ScadaBridge:Security:JwtSigningKey` | HMAC-SHA256 signing key for the cookie-embedded session JWT. | ≥32-char secret | Yes on Central |
| `ScadaBridge__Security__Ldap__ServiceAccountPassword` | Yes — `${secret:ldap/scadabridge/service-account-password}` in `appsettings.Central.json` | `ScadaBridge:Security:Ldap:ServiceAccountPassword` | LDAP bind-account password. (Renamed from the old flat `...__LdapServiceAccountPassword`.) | the bind password | Yes on Central if LDAP bind is used |
| `ScadaBridge__InboundApi__ApiKeyPepper` | No — env/secret only (not in any shipped appsettings) | `ScadaBridge:InboundApi:ApiKeyPepper` | Per-environment pepper for the inbound API-key peppered-HMAC verifier. Hard Central startup requirement since 2026-06-03. | ≥16-char secret, unique per environment | Yes on Central |
@@ -164,7 +163,7 @@ ScadaBridge code — included for completeness when standing up a rig.
**Central node** (via env, or the `${secret:...}` equivalents + `ZB_SECRETS_MASTER_KEY`):
`SCADABRIDGE_CONFIG=Central`, `ScadaBridge__Database__ConfigurationDb`,
`ScadaBridge__Database__MachineDataDb`, `ScadaBridge__Security__JwtSigningKey`,
`ScadaBridge__Security__JwtSigningKey`,
`ScadaBridge__Security__Ldap__ServiceAccountPassword`, `ScadaBridge__InboundApi__ApiKeyPepper`, and
for each managed site either the `SB-GRPC-PSK-<siteId>` secret or `ScadaBridge__Communication__SitePsks__<siteId>`.
@@ -0,0 +1,125 @@
using System.Data.Common;
using System.Net.Sockets;
using Microsoft.Data.SqlClient;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <summary>
/// Classifies a central audit-insert failure (review 2026-10-21 06/IMPL-1 follow-up) so that a
/// DATABASE problem — a maintenance window, a timeout, a dropped connection, an exhausted budget —
/// is never mistaken for a verdict about the ROW. Two questions, two predicates:
/// <list type="bullet">
/// <item><see cref="IsPermanentRowRejection"/> — narrow: the row itself can never be inserted
/// (string truncation, NULL into NOT NULL, a conversion / overflow, a constraint). Only these ids
/// go back to the site as <c>rejected_event_ids</c>, which is what lets the site give a row up as
/// <c>Rejected</c>.</item>
/// <item><see cref="IsTransient"/> — broad: anything that may clear on its own. The reconciliation
/// puller never counts these toward permanently abandoning a row.</item>
/// </list>
/// The exception's whole <see cref="Exception.InnerException"/> chain is inspected, because EF Core
/// and the execution strategy wrap the provider exception.
/// </summary>
internal static class AuditIngestErrorClassifier
{
/// <summary>
/// SQL Server errors that describe the ROW, not the server: 8152 / 2628 string or binary data
/// would be truncated, 515 NULL into a NOT NULL column, 245 / 241 / 242 / 8114 / 8115 / 220 / 232
/// conversion and arithmetic overflow, 547 CHECK / FOREIGN KEY constraint.
/// </summary>
private static readonly HashSet<int> PermanentRowErrorNumbers = new()
{
8152, 2628, 515, 245, 241, 242, 8114, 8115, 220, 232, 547,
};
/// <summary>
/// SQL Server errors that may clear on their own (timeout, connection loss, deadlock, lock
/// timeout, resource / Azure throttling, database unavailable or in a restricted state).
/// </summary>
private static readonly HashSet<int> TransientErrorNumbers = new()
{
-2, -1, 0, 2, 20, 53, 64, 121, 233, 1205, 1222, 3960, 4060, 4221, 8645, 8651,
10053, 10054, 10060, 10928, 10929, 40143, 40197, 40501, 40540, 40613,
49918, 49919, 49920, 942, 952, 983, 976, 978, 3906, 9001, 9002,
};
/// <summary>
/// True when the failure is a SQL Server error that says the row itself can never be inserted.
/// Everything else — including every transient error and every non-SQL exception — is false,
/// so the site keeps such rows Pending and simply retries them.
/// </summary>
/// <param name="ex">The insert failure.</param>
/// <returns>Whether the row should be reported back to the site as permanently rejected.</returns>
public static bool IsPermanentRowRejection(Exception ex)
{
if (IsTransient(ex))
{
return false;
}
for (var e = ex; e is not null; e = e.InnerException)
{
if (e is SqlException sql)
{
foreach (SqlError error in sql.Errors)
{
if (PermanentRowErrorNumbers.Contains(error.Number))
{
return true;
}
}
return PermanentRowErrorNumbers.Contains(sql.Number);
}
}
return false;
}
/// <summary>
/// True when any exception in the chain is a timeout, cancellation, socket / IO / connection
/// failure, an EF retry-limit exhaustion, a provider error flagged transient, or a SQL Server
/// error number in the transient set.
/// </summary>
/// <param name="ex">The insert failure.</param>
/// <returns>Whether the failure may clear on its own.</returns>
public static bool IsTransient(Exception ex)
{
for (var e = ex; e is not null; e = e.InnerException)
{
switch (e)
{
case TimeoutException:
case OperationCanceledException:
case SocketException:
case IOException:
case HttpRequestException:
return true;
case SqlException sql:
if (TransientErrorNumbers.Contains(sql.Number))
{
return true;
}
foreach (SqlError error in sql.Errors)
{
if (TransientErrorNumbers.Contains(error.Number))
{
return true;
}
}
if (sql.IsTransient)
{
return true;
}
break;
case DbException db when db.IsTransient:
return true;
}
// EF Core's RetryLimitExceededException (an InvalidOperationException) wraps a
// transient provider error once the execution strategy gives up.
if (e.GetType().Name == "RetryLimitExceededException")
{
return true;
}
}
return false;
}
}
@@ -152,6 +152,7 @@ public class AuditLogIngestActor : ReceiveActor
var replyTo = Sender;
var nowUtc = DateTime.UtcNow;
var accepted = new List<Guid>(cmd.Events.Count);
var rejected = new List<Guid>();
// Resolve the repository for the whole batch — one DbContext per
// message, mirroring NotificationOutboxActor.
@@ -184,7 +185,7 @@ public class AuditLogIngestActor : ReceiveActor
// unregistered (test composition roots) means the per-row catch
// simply logs without surfacing on the health dashboard.
var failureCounter = scope.ServiceProvider.GetService<ICentralAuditWriteFailureCounter>();
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted)
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted, rejected)
.ConfigureAwait(false);
}
catch (Exception ex)
@@ -201,7 +202,7 @@ public class AuditLogIngestActor : ReceiveActor
accepted.Count);
}
replyTo.Tell(new IngestAuditEventsReply(accepted));
replyTo.Tell(new IngestAuditEventsReply(accepted) { RejectedEventIds = rejected });
}
private async Task IngestWithRepositoryAsync(
@@ -210,7 +211,8 @@ public class AuditLogIngestActor : ReceiveActor
ICentralAuditWriteFailureCounter? failureCounter,
IngestAuditEventsCommand cmd,
DateTime nowUtc,
List<Guid> accepted)
List<Guid> accepted,
List<Guid> rejected)
{
// Stamp the ingest timestamp here, not at the site. Redact BEFORE the
// IngestedAtUtc stamp so the redacted copy carries the central-side
@@ -227,7 +229,7 @@ public class AuditLogIngestActor : ReceiveActor
projected.Add(AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc));
}
// ONE set-based statement for the whole packet (WP2.2) instead of one
// ONE set-based statement for the whole packet instead of one
// IF NOT EXISTS … INSERT round trip per event. A telemetry packet of 200
// rows used to cost 200 sequential round trips on the central singleton's
// dispatcher; it now costs two. Idempotency is unchanged — duplicates
@@ -303,6 +305,15 @@ public class AuditLogIngestActor : ReceiveActor
catch { /* counter must never throw — defence in depth */ }
}
// Review 2026-10-21 06/IMPL-1: tell the site ONLY about rows the server
// refused for a reason intrinsic to the row (truncation, NOT NULL, conversion,
// constraint). A timeout / connection loss / maintenance window leaves the id
// in neither list, so the site retries it without counting a refusal.
if (AuditIngestErrorClassifier.IsPermanentRowRejection(rowEx))
{
rejected.Add(cmd.Events[i].EventId);
}
_logger.LogError(rowEx,
"Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.",
cmd.Events[i].EventId);
@@ -336,6 +347,7 @@ public class AuditLogIngestActor : ReceiveActor
{
var replyTo = Sender;
var accepted = new List<Guid>(cmd.Entries.Count);
var rejected = new List<Guid>();
try
{
@@ -356,7 +368,7 @@ public class AuditLogIngestActor : ReceiveActor
var strategy = dbContext.Database.CreateExecutionStrategy();
// Fast path (WP2.2): the WHOLE packet in ONE transaction — one
// Fast path: the WHOLE packet in ONE transaction — one
// set-based audit insert plus one single-statement upsert per entry.
// The predecessor opened a transaction PER entry and issued three
// statements inside it (IF NOT EXISTS insert, then the two-statement
@@ -370,7 +382,7 @@ public class AuditLogIngestActor : ReceiveActor
strategy, dbContext, auditRepo, siteCallRepo, redactor, cmd, accepted, budget.Token)
.ConfigureAwait(false))
{
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
replyTo.Tell(new IngestCachedTelemetryReply(accepted) { RejectedEventIds = rejected });
return;
}
@@ -423,7 +435,7 @@ public class AuditLogIngestActor : ReceiveActor
var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance;
var filteredAudit = safeRedactor.Apply(entry.Audit);
var auditStamped = AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt);
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
var siteCallStamped = SiteCallColumnBounds.Clamp(entry.SiteCall) with { IngestedAtUtc = ingestedAt };
await auditRepo.InsertIfNotExistsAsync(auditStamped, fallbackBudget.Token)
.ConfigureAwait(false);
@@ -445,6 +457,11 @@ public class AuditLogIngestActor : ReceiveActor
// sustained dual-write failure surfaces on the dashboard.
try { failureCounter?.Increment(); }
catch { /* counter must never throw — defence in depth */ }
if (AuditIngestErrorClassifier.IsPermanentRowRejection(ex))
{
rejected.Add(entry.Audit.EventId);
}
_logger.LogError(
ex,
"Combined telemetry dual-write failed for AuditEvent {EventId} / TrackedOperationId {TrackedOpId}; rolled back.",
@@ -463,7 +480,7 @@ public class AuditLogIngestActor : ReceiveActor
"Combined telemetry batch ingest failed before per-entry processing.");
}
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
replyTo.Tell(new IngestCachedTelemetryReply(accepted) { RejectedEventIds = rejected });
}
/// <summary>
@@ -522,7 +539,7 @@ public class AuditLogIngestActor : ReceiveActor
foreach (var entry in cmd.Entries)
{
await siteCallRepo
.UpsertAsync(entry.SiteCall with { IngestedAtUtc = ingestedAt }, ct)
.UpsertAsync(SiteCallColumnBounds.Clamp(entry.SiteCall) with { IngestedAtUtc = ingestedAt }, ct)
.ConfigureAwait(false);
}
@@ -18,8 +18,8 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// than <c>DateTime.UtcNow - RetentionDays</c>.</item>
/// <item>For each eligible boundary, calls
/// <see cref="IAuditLogRepository.SwitchOutPartitionAsync"/>, a
/// metadata-only staging-table switch (WP2.2 removed the index
/// drop/rebuild that used to bracket it).</item>
/// metadata-only staging-table switch (no index
/// drop/rebuild brackets it).</item>
/// <item>Publishes <see cref="AuditLogPurgedEvent"/> on the actor-system
/// EventStream so the central health collector + ops surfaces
/// can subscribe without coupling to this actor.</item>
@@ -11,8 +11,8 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <remarks>
/// <para>
/// The purge actor is a daily-cadence singleton, not a hot-loop. The
/// partition switch itself is metadata-only (since WP2.2's
/// <c>AlignAuditLogEventIdUniqueness</c> it no longer drops and rebuilds a
/// partition switch itself is metadata-only (since the
/// <c>AlignAuditLogEventIdUniqueness</c> migration it no longer drops and rebuilds a
/// non-aligned unique index around it), but it still takes schema-modification
/// locks and competes with ingest for the same table. Lower this only when an
/// operator can prove they need sub-daily purge granularity.
@@ -0,0 +1,66 @@
using System.Collections.Concurrent;
using Grpc.Net.Client;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <summary>
/// Optional capability of a pull invoker that caches one <see cref="GrpcChannel"/> per
/// <c>(site, endpoint)</c>: drop (and dispose) every cached channel whose key is not in the
/// current site enumeration. The pull clients call it with each enumeration they already make
/// to resolve an endpoint, so a deleted site or an edited address no longer leaves a dead
/// channel (and its HTTP/2 connection pool) alive until process exit (review 2026-10-21 06/MEM-2).
/// </summary>
public interface IGrpcChannelCachePruner
{
/// <summary>Retains only channels whose <c>(site, endpoint)</c> appears in <paramref name="liveSites"/>.</summary>
/// <param name="liveSites">The current site enumeration.</param>
void RetainOnly(IReadOnlyList<SiteEntry> liveSites);
}
/// <summary>Shared implementation of <see cref="IGrpcChannelCachePruner.RetainOnly"/>.</summary>
internal static class GrpcChannelCachePruning
{
/// <summary>
/// Removes and disposes every channel in <paramref name="channels"/> whose key is neither a
/// live site's primary nor its fallback endpoint.
/// </summary>
/// <param name="channels">The invoker's channel cache.</param>
/// <param name="liveSites">The current site enumeration.</param>
/// <returns>The number of channels pruned.</returns>
public static int RetainOnly(
ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> channels,
IReadOnlyList<SiteEntry> liveSites)
{
if (channels.IsEmpty)
{
return 0;
}
var live = new HashSet<(string, string)>();
foreach (var site in liveSites)
{
if (!string.IsNullOrWhiteSpace(site.GrpcEndpoint))
{
live.Add((site.SiteId, site.GrpcEndpoint));
}
if (!string.IsNullOrWhiteSpace(site.FallbackGrpcEndpoint))
{
live.Add((site.SiteId, site.FallbackGrpcEndpoint!));
}
}
var pruned = 0;
foreach (var key in channels.Keys)
{
if (!live.Contains(key) && channels.TryRemove(key, out var stale))
{
// A call racing this on the stale channel fails as a tolerable transport
// fault and the next tick re-resolves — acceptable for an address that is
// no longer configured.
stale.Dispose();
pruned++;
}
}
return pruned;
}
}
@@ -194,6 +194,8 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
string siteId, CancellationToken ct)
{
var sites = await _sites.EnumerateAsync(ct).ConfigureAwait(false);
// Review 2026-10-21 06/MEM-2: drop channels for sites/addresses no longer enumerated.
(_invoker as IGrpcChannelCachePruner)?.RetainOnly(sites);
foreach (var site in sites)
{
if (string.Equals(site.SiteId, siteId, StringComparison.Ordinal) &&
@@ -256,14 +258,13 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
/// endpoint string, so a changed site address (NodeA→NodeB failover flip / an
/// edited gRPC address) is reached as soon as the resolver hands the new
/// endpoint to <see cref="InvokeAsync"/> — it creates a fresh channel for the
/// new address. Unlike <c>SiteStreamGrpcClientFactory</c> (keyed by siteId,
/// which actively evicts a re-keyed client), the channel for the previous
/// address is NOT actively evicted here; it lingers idle until
/// <see cref="Dispose"/>. Idle channels hold no streams, so this is a minor
/// cache footprint cost, not a correctness or liveness gap.
/// new address. The channel for a previous address (or a deleted site) is
/// evicted and disposed by <see cref="RetainOnly"/>, which the client calls with
/// every site enumeration it already makes (review 2026-10-21 06/MEM-2) — before
/// that it lingered, with its HTTP/2 connection pool, until process exit.
/// </summary>
public sealed class GrpcPullAuditEventsInvoker
: GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable
: GrpcPullAuditEventsClient.IPullAuditEventsInvoker, IDisposable, IGrpcChannelCachePruner
{
private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options;
@@ -344,6 +345,10 @@ public sealed class GrpcPullAuditEventsInvoker
},
}.WithMessageLimits().WithSiteCredentials(_pskProvider, siteId));
/// <inheritdoc />
public void RetainOnly(IReadOnlyList<SiteEntry> liveSites) =>
GrpcChannelCachePruning.RetainOnly(_channels, liveSites);
/// <summary>Disposes all cached channels.</summary>
public void Dispose()
{
@@ -141,7 +141,9 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
{
try
{
var sc = SiteCallDtoMapper.FromDto(dto) with { SourceSite = siteId };
// Fit LastError to the central column so the reconciliation upsert cannot fail
// with SQL 8152 (review 2026-10-21 follow-up — see SiteCallColumnBounds).
var sc = SiteCallColumnBounds.Clamp(SiteCallDtoMapper.FromDto(dto)) with { SourceSite = siteId };
siteCalls.Add(sc);
}
catch (Exception ex)
@@ -214,6 +216,8 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
string siteId, CancellationToken ct)
{
var sites = await _sites.EnumerateAsync(ct).ConfigureAwait(false);
// Review 2026-10-21 06/MEM-2: drop channels for sites/addresses no longer enumerated.
(_invoker as IGrpcChannelCachePruner)?.RetainOnly(sites);
foreach (var site in sites)
{
if (string.Equals(site.SiteId, siteId, StringComparison.Ordinal) &&
@@ -275,13 +279,13 @@ public sealed class GrpcPullSiteCallsClient : IPullSiteCallsClient
/// issues the unary <c>PullSiteCallsAsync</c> call. The cache is keyed by
/// endpoint string, so a changed site address (NodeA→NodeB failover flip / an
/// edited gRPC address) is reached as soon as the resolver hands the new endpoint
/// to <see cref="InvokeAsync"/>. The channel for a previous address lingers idle
/// until <see cref="Dispose"/> (idle channels hold no streams — a minor cache
/// footprint cost, not a correctness or liveness gap). Sibling of
/// to <see cref="InvokeAsync"/>. The channel for a previous address (or a deleted
/// site) is evicted and disposed by <see cref="RetainOnly"/>, which the client calls
/// with every site enumeration (review 2026-10-21 06/MEM-2). Sibling of
/// <see cref="GrpcPullAuditEventsInvoker"/>.
/// </summary>
public sealed class GrpcPullSiteCallsInvoker
: GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable
: GrpcPullSiteCallsClient.IPullSiteCallsInvoker, IDisposable, IGrpcChannelCachePruner
{
private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options;
@@ -358,6 +362,10 @@ public sealed class GrpcPullSiteCallsInvoker
},
}.WithMessageLimits().WithSiteCredentials(_pskProvider, siteId));
/// <inheritdoc />
public void RetainOnly(IReadOnlyList<SiteEntry> liveSites) =>
GrpcChannelCachePruning.RetainOnly(_channels, liveSites);
/// <summary>Disposes all cached channels.</summary>
public void Dispose()
{
@@ -28,6 +28,9 @@ public sealed class GrpcPullTraceEventsClient : IPullTraceEventsClient
private readonly ILogger<GrpcPullTraceEventsClient> _logger;
/// <summary>Creates the client over the given site enumerator and unary-call invoker.</summary>
/// <param name="sites">Enumerates the sites and their gRPC endpoints.</param>
/// <param name="invoker">Issues the unary <c>PullTraceEvents</c> call.</param>
/// <param name="logger">Logger for pull diagnostics.</param>
public GrpcPullTraceEventsClient(
ISiteEnumerator sites,
IPullTraceEventsInvoker invoker,
@@ -132,6 +135,8 @@ public sealed class GrpcPullTraceEventsClient : IPullTraceEventsClient
private async Task<(string? Primary, string? Fallback)> ResolveEndpointsAsync(string siteId, CancellationToken ct)
{
var sites = await _sites.EnumerateAsync(ct).ConfigureAwait(false);
// Review 2026-10-21 06/MEM-2: drop channels for sites/addresses no longer enumerated.
(_invoker as IGrpcChannelCachePruner)?.RetainOnly(sites);
foreach (var site in sites)
{
if (string.Equals(site.SiteId, siteId, StringComparison.Ordinal) && !string.IsNullOrWhiteSpace(site.GrpcEndpoint))
@@ -152,6 +157,11 @@ public sealed class GrpcPullTraceEventsClient : IPullTraceEventsClient
public interface IPullTraceEventsInvoker
{
/// <summary>Issues the RPC against <paramref name="endpoint"/> with the site's preshared key.</summary>
/// <param name="siteId">The site the call targets.</param>
/// <param name="endpoint">The site node gRPC endpoint to call.</param>
/// <param name="request">The pull request.</param>
/// <param name="ct">Token to cancel the call.</param>
/// <returns>The site's trace-event response.</returns>
Task<PullTraceEventsResponse> InvokeAsync(
string siteId, string endpoint, PullTraceEventsRequest request, CancellationToken ct);
}
@@ -162,13 +172,15 @@ public sealed class GrpcPullTraceEventsClient : IPullTraceEventsClient
/// <see cref="GrpcChannel"/> per (site, endpoint), keepalive from <see cref="CommunicationOptions"/>,
/// per-site PSK call credentials — the same shape as <see cref="GrpcPullAuditEventsInvoker"/>.
/// </summary>
public sealed class GrpcPullTraceEventsInvoker : GrpcPullTraceEventsClient.IPullTraceEventsInvoker, IDisposable
public sealed class GrpcPullTraceEventsInvoker : GrpcPullTraceEventsClient.IPullTraceEventsInvoker, IDisposable, IGrpcChannelCachePruner
{
private readonly ConcurrentDictionary<(string Site, string Endpoint), GrpcChannel> _channels = new();
private readonly CommunicationOptions _options;
private readonly ISitePskProvider? _pskProvider;
/// <summary>Creates the invoker with the given options and optional per-site PSK provider.</summary>
/// <param name="options">Communication options supplying keepalive settings.</param>
/// <param name="pskProvider">Optional provider of per-site preshared keys.</param>
public GrpcPullTraceEventsInvoker(CommunicationOptions options, ISitePskProvider? pskProvider)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
@@ -213,6 +225,10 @@ public sealed class GrpcPullTraceEventsInvoker : GrpcPullTraceEventsClient.IPull
}.WithMessageLimits().WithSiteCredentials(_pskProvider, siteId));
/// <inheritdoc />
public void RetainOnly(IReadOnlyList<SiteEntry> liveSites) =>
GrpcChannelCachePruning.RetainOnly(_channels, liveSites);
/// <summary>Disposes every cached gRPC channel.</summary>
public void Dispose()
{
foreach (var channel in _channels.Values)
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
@@ -24,8 +25,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// (<see cref="ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteAuditTelemetryActor"/> +
/// <c>IngestAuditEvents</c>) is the primary mechanism. This actor exists so a
/// missed push (gRPC blip, central restart, site offline) is eventually
/// repaired by central re-pulling whatever the site still has in
/// <c>Pending</c>/<c>Forwarded</c> state. Idempotency on
/// repaired by central re-pulling whatever the site has NOT yet had acknowledged —
/// <c>Pending</c> rows plus rows that left the push queue without an ack
/// (<c>PushAbandoned</c>, <c>Rejected</c>). <c>Forwarded</c> rows are no longer
/// re-served: central already acked them, and re-shipping every pushed row capped the
/// pull at a throughput no busy site could keep up with (review 2026-10-21 06/PERF-1).
/// A pre-PERF-1 site still serves them; idempotency on
/// <see cref="AuditEvent.EventId"/> makes duplicate
/// arrivals from both paths a silent no-op.
/// </para>
@@ -34,15 +39,18 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// kept in-memory for the actor's lifetime. The cluster singleton normally
/// survives the host process; on a deliberate failover OR a singleton restart
/// the cursors reset to <see cref="DateTime.MinValue"/>. That is conservative
/// but correct — the next tick simply asks for everything the site still has,
/// and idempotent ingest swallows the dupes. Persisting cursors to MS SQL was
/// but correct — the next tick simply asks for every row the site still owes,
/// and idempotent ingest swallows the dupes. Because the site serves only
/// unacknowledged rows, that re-pull is the small owed set, not a multi-day
/// replay of every forwarded row (review 2026-10-21 06/COR-3). Persisting cursors to MS SQL was
/// considered and rejected: the cost of a write per tick outweighs the
/// rare benefit of avoiding one over-broad pull after a restart.
/// </para>
/// <para>
/// <b>Stalled detection.</b> The brief calls a site "stalled" when two
/// consecutive pull cycles BOTH return non-empty AND <c>MoreAvailable=true</c>
/// — i.e. the backlog isn't draining. The actor publishes
/// <b>Stalled detection.</b> Each tick pages through a site (up to
/// <see cref="SiteAuditReconciliationOptions.MaxPagesPerTick"/> pulls, each ingested with
/// one set-based insert). A site is "stalled" when two consecutive ticks BOTH still end
/// non-empty AND <c>MoreAvailable=true</c> after that — i.e. the backlog isn't draining. The actor publishes
/// <see cref="SiteAuditTelemetryStalledChanged"/> on the actor system's
/// EventStream so a future <c>ICentralHealthCollector</c> bridge
/// can flip the health metric without coupling this actor to the health
@@ -173,6 +181,52 @@ public class SiteAuditReconciliationActor : ReceiveActor
private ICancelable? _timer;
/// <summary>
/// Upper bound on <see cref="_abandonedEventIds"/>. The set is only a short-circuit (see its
/// doc comment — determinism of <see cref="DeriveAbandonmentEventId"/> is what makes
/// re-abandonment safe), so clearing it when it grows past this costs at most one re-run of
/// the retry cycle per forgotten row, never a duplicate or a lost row (review 2026-10-21
/// 06/MEM-2: it used to grow one entry per abandoned row for the singleton's lifetime).
/// </summary>
internal const int MaxRememberedAbandonedEventIds = 10_000;
internal void RememberAbandoned(Guid eventId)
{
if (_abandonedEventIds.Count >= MaxRememberedAbandonedEventIds)
{
_abandonedEventIds.Clear();
}
_abandonedEventIds.Add(eventId);
}
/// <summary>Test seam: current size of the abandoned-id short-circuit set.</summary>
internal int AbandonedEventIdCountForTests => _abandonedEventIds.Count;
/// <summary>
/// Drops per-site state (cursor, non-draining counter, stalled latch) for sites no longer in
/// the enumeration — a deleted or renamed site otherwise kept its entries for the singleton's
/// lifetime (review 2026-10-21 06/MEM-2, mirroring <c>CentralHealthAggregator.PruneUnknownSites</c>).
/// A pruned site that was latched stalled publishes <c>Stalled=false</c> so the health
/// surface does not keep a badge for a site that no longer exists.
/// </summary>
private void PruneUnknownSites(IReadOnlyList<SiteEntry> sites, Akka.Event.EventStream eventStream)
{
var live = new HashSet<string>(sites.Select(s => s.SiteId), StringComparer.Ordinal);
foreach (var siteId in _cursors.Keys.Concat(_nonDrainingCycles.Keys).Concat(_stalled.Keys)
.Where(id => !live.Contains(id)).Distinct().ToList())
{
_cursors.Remove(siteId);
_nonDrainingCycles.Remove(siteId);
if (_stalled.Remove(siteId, out var wasStalled) && wasStalled)
{
eventStream.Publish(new SiteAuditTelemetryStalledChanged(siteId, Stalled: false));
}
}
}
/// <summary>Test seam: number of sites with a held reconciliation cursor.</summary>
internal int TrackedSiteCountForTests => _cursors.Count;
/// <summary>
/// Initializes the reconciliation actor with its dependencies and registers the tick handler.
/// </summary>
@@ -253,6 +307,10 @@ public class SiteAuditReconciliationActor : ReceiveActor
return;
}
// After the empty guard: an empty enumeration is as likely a transient lookup glitch
// as a fleet with no sites, and pruning on it would throw away every cursor.
PruneUnknownSites(sites, eventStream);
// Use CreateAsyncScope + await using so scoped EF Core
// services (IAsyncDisposable DbContexts) dispose asynchronously
// without blocking on sync Dispose() of pending connection cleanup.
@@ -288,23 +346,64 @@ public class SiteAuditReconciliationActor : ReceiveActor
}
}
/// <summary>
/// Drains one site for this tick: issues up to
/// <see cref="SiteAuditReconciliationOptions.MaxPagesPerTick"/> <c>PullAuditEvents</c>
/// RPCs back to back while the site reports <c>MoreAvailable</c> and the cursor keeps
/// advancing, then updates the stalled signal from the LAST page.
/// </summary>
/// <remarks>
/// Paging within a tick (review 2026-10-21 06/PERF-1, mirroring <c>SiteCallAuditActor</c>'s
/// ≤50-page loop) is what makes the stalled latch mean something: one pull per tick capped
/// the pull at <c>BatchSize</c> rows per <c>ReconciliationInterval</c> (0.85 rows/s at the
/// defaults), so any busy-but-healthy site reported <c>MoreAvailable</c> on every tick and
/// latched <c>SiteAuditTelemetryStalled</c> permanently. Now a site is non-draining only when
/// it still has more rows after the page cap, or its cursor is held back by a row that keeps
/// failing — a genuine non-drain.
/// </remarks>
private async Task PullSiteAsync(SiteEntry site, IAuditLogRepository repository, Akka.Event.EventStream eventStream)
{
var maxPages = Math.Max(1, _options.MaxPagesPerTick);
var nonDraining = false;
for (var page = 0; page < maxPages; page++)
{
var (response, cursorAdvanced) = await PullPageAsync(site, repository).ConfigureAwait(false);
nonDraining = response.MoreAvailable && response.Events.Count > 0;
if (!nonDraining || !cursorAdvanced)
{
break;
}
}
UpdateStalledState(site.SiteId, draining: !nonDraining, eventStream);
}
/// <summary>
/// Issues one <c>PullAuditEvents</c> RPC against the site, ingests the
/// returned rows idempotently into the central repository, and advances
/// the composite <c>(<see cref="AuditEvent.OccurredAtUtc"/>, <see cref="AuditEvent.EventId"/>)</c>
/// cursor to the maximum row observed. The brief's "saturate until backlog clears" intent is
/// met by the natural cadence — each tick issues one pull, and a backed-up site
/// drains across consecutive ticks. The stalled signal (two non-draining
/// ticks in a row) surfaces when that drain isn't keeping up.
/// cursor to the maximum row observed.
/// </summary>
/// <remarks>
/// Unlike <c>SiteCallAuditActor</c> this does NOT page within a tick; one pull per tick is
/// deliberate (the stalled signal is how a lagging drain surfaces). The composite cursor is
/// what makes that safe against a same-instant burst larger than one batch: the id tiebreak
/// advances even when the timestamp cannot, so consecutive ticks make real progress instead
/// of re-serving the same window forever.
/// The page is ingested with ONE set-based
/// <see cref="IAuditLogRepository.InsertManyIfNotExistsAsync"/> call (review 2026-10-21
/// 06/PERF-1 — it used to cost one <c>InsertIfNotExistsAsync</c> round trip per row). Only
/// when the batch throws does the page fall back to the per-row path, which keeps the
/// documented poison-row isolation: the failing row is retried up to
/// <see cref="MaxPermanentInsertAttempts"/> times with the cursor held back, then
/// permanently abandoned. Only a row-intrinsic failure
/// (<see cref="AuditIngestErrorClassifier.IsPermanentRowRejection"/>) counts toward that
/// threshold, and a row is abandoned only on a tick where at least one sibling row of the same
/// page inserted — a page where every row fails is a database problem (schema drift after a
/// missed migration, a narrowed column), and abandoning it would let the site mark every row
/// <c>Reconciled</c> and purge it (review 2026-10-21 follow-up M1). Re-running rows the failed batch may already have committed is
/// safe — every insert is idempotent on <see cref="AuditEvent.EventId"/>, which is also
/// what lets a pre-PERF-1 site that still re-serves <c>Forwarded</c> rows be pulled
/// unchanged.
/// </remarks>
private async Task PullSiteAsync(SiteEntry site, IAuditLogRepository repository, Akka.Event.EventStream eventStream)
/// <returns>The site's response and whether the cursor moved.</returns>
private async Task<(PullAuditEventsResponse Response, bool CursorAdvanced)> PullPageAsync(
SiteEntry site, IAuditLogRepository repository)
{
var cursor = _cursors.TryGetValue(site.SiteId, out var c)
? c
@@ -320,6 +419,74 @@ public class SiteAuditReconciliationActor : ReceiveActor
var maxAfterId = afterId;
var hasUnresolvedFailure = false;
var nowUtc = DateTime.UtcNow;
// Set-based fast path for every row not already known-abandoned.
var batchCommitted = false;
var toInsert = new List<ZB.MOM.WW.Audit.AuditEvent>(response.Events.Count);
foreach (var evt in response.Events)
{
if (!_abandonedEventIds.Contains(evt.EventId))
{
toInsert.Add(AuditRowProjection.WithIngestedAtUtc(evt, nowUtc));
}
}
if (toInsert.Count == 0)
{
batchCommitted = true;
}
else
{
try
{
await repository.InsertManyIfNotExistsAsync(toInsert).ConfigureAwait(false);
batchCommitted = true;
foreach (var evt in toInsert)
{
_failedInsertAttempts.Remove(evt.EventId);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Set-based reconciliation ingest of {Count} audit event(s) from site {SiteId} failed; falling back to per-row inserts so one bad row does not sink the page.",
toInsert.Count, site.SiteId);
}
}
// Per-row fallback (only when the set-based insert threw). Every row is attempted FIRST and
// its outcome recorded, so the abandonment decision below can see whether ANY sibling in
// this page inserted (review 2026-10-21 follow-up M1).
var rowFailures = new Dictionary<Guid, Exception>();
var anyRowInserted = false;
if (!batchCommitted)
{
foreach (var evt in response.Events)
{
if (_abandonedEventIds.Contains(evt.EventId))
{
continue;
}
try
{
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
anyRowInserted = true;
}
catch (Exception ex)
{
rowFailures[evt.EventId] = ex;
}
}
}
foreach (var evt in response.Events)
{
var advanceForThisRow = false;
@@ -335,33 +502,42 @@ public class SiteAuditReconciliationActor : ReceiveActor
// row collapse into the existing one anyway.
advanceForThisRow = true;
}
else if (batchCommitted || !rowFailures.TryGetValue(evt.EventId, out var ex))
{
advanceForThisRow = true;
}
else
{
try
// Per-row verdict so one bad event does not abandon the rest of the batch.
// Only a ROW-INTRINSIC failure (AuditIngestErrorClassifier.IsPermanentRowRejection
// — the same narrow set the push path reports back as rejected) counts toward
// MaxPermanentInsertAttempts; below the threshold the cursor is HELD BACK so the
// next tick re-pulls and retries; at the threshold the row is permanently
// abandoned (LogCritical + cursor advances past) to keep a truly broken row from
// blocking all subsequent progress for the site.
if (!AuditIngestErrorClassifier.IsPermanentRowRejection(ex))
{
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
advanceForThisRow = true;
// A database problem (timeout, connection loss, deadlock, throttling, a
// maintenance window) or anything not provably about THIS row — e.g. schema
// drift after a missed central migration (207 / 208 invalid column/object) —
// is not a verdict about the row: hold the cursor and retry, but never count
// it toward permanent abandonment. Abandoning such rows flips the site's
// cursor past them, the site marks them Reconciled, and its retention purge
// deletes them — a silent loss of every row of the outage (review 2026-10-21
// 06/PERF-1 adversarial follow-up; follow-up M1).
_logger.LogWarning(
ex,
"Reconciliation ingest of AuditEvent {EventId} from site {SiteId} failed with a non-row-specific error; cursor held back, not counted toward abandonment.",
evt.EventId,
site.SiteId);
hasUnresolvedFailure = true;
}
catch (Exception ex)
else
{
// Per-row catch so one bad event does not abandon
// the rest of the batch. Track the failure count per EventId —
// below MaxPermanentInsertAttempts the cursor is HELD BACK so
// the next tick re-pulls and retries; at the threshold the row
// is permanently abandoned (LogCritical + cursor advances past)
// to keep a truly broken row from blocking all subsequent
// progress for the site.
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
_failedInsertAttempts[evt.EventId] = attempts;
if (attempts >= MaxPermanentInsertAttempts)
if (attempts >= MaxPermanentInsertAttempts && anyRowInserted)
{
_logger.LogCritical(
ex,
@@ -370,7 +546,7 @@ public class SiteAuditReconciliationActor : ReceiveActor
site.SiteId,
attempts);
_failedInsertAttempts.Remove(evt.EventId);
_abandonedEventIds.Add(evt.EventId);
RememberAbandoned(evt.EventId);
advanceForThisRow = true;
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
@@ -378,13 +554,19 @@ public class SiteAuditReconciliationActor : ReceiveActor
}
else
{
// At the threshold but no sibling in this page inserted: every row failing
// identically is a DATABASE problem (e.g. a column narrowed by a migration
// that 245/8114s every row), not one poison row — hold, never abandon. The
// row is abandoned on a later tick whose page also carries a row that
// inserts, which proves the database itself accepts writes.
_logger.LogError(
ex,
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}{NoSibling}); cursor held back for retry.",
evt.EventId,
site.SiteId,
attempts,
MaxPermanentInsertAttempts);
MaxPermanentInsertAttempts,
anyRowInserted ? string.Empty : ", no sibling row inserted this tick");
hasUnresolvedFailure = true;
}
}
@@ -413,12 +595,13 @@ public class SiteAuditReconciliationActor : ReceiveActor
// another attempt. Once it succeeds (or hits the permanent-abandon
// threshold) the cursor unblocks naturally. Both halves move together: holding the
// timestamp back while advancing the id would skip the very rows being retried.
_cursors[site.SiteId] = hasUnresolvedFailure
var next = hasUnresolvedFailure
? (since, afterId)
: (maxOccurred, maxAfterId);
_cursors[site.SiteId] = next;
var nonDraining = response.MoreAvailable && response.Events.Count > 0;
UpdateStalledState(site.SiteId, draining: !nonDraining, eventStream);
var advanced = next.Item1 != since || !string.Equals(next.Item2, afterId, StringComparison.Ordinal);
return (response, advanced);
}
/// <summary>
@@ -521,6 +704,8 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// lost rows get distinct synthetic ids.
/// </para>
/// </remarks>
/// <param name="lostEventId">The event id of the lost row being abandoned.</param>
/// <returns>A deterministic synthetic GUID for the abandonment row.</returns>
internal static Guid DeriveAbandonmentEventId(Guid lostEventId)
{
Span<byte> input = stackalloc byte[32];
@@ -3,8 +3,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <summary>
/// Tuning knobs for the central <see cref="SiteAuditReconciliationActor"/> singleton.
/// Defaults: pull every 5 minutes per site, 256 rows per
/// batch, declare a site "stalled" after two consecutive pull cycles return non-empty
/// AND <c>MoreAvailable=true</c> (the backlog is not draining).
/// page and up to 50 pages per site per tick, declare a site "stalled" after two
/// consecutive cycles still end non-empty AND <c>MoreAvailable=true</c> after the page cap
/// (the backlog is not draining).
/// </summary>
/// <remarks>
/// <para>
@@ -77,6 +78,15 @@ public sealed class SiteAuditReconciliationOptions
/// </summary>
public int BatchSize { get; set; } = 256;
/// <summary>
/// Maximum <c>PullAuditEvents</c> pages issued against one site in one tick while the
/// site keeps reporting <c>MoreAvailable</c> (review 2026-10-21 06/PERF-1, mirroring
/// <c>SiteCallAuditActor</c>'s 50-page bound). Bounds a misbehaving site's per-tick cost
/// while letting a busy-but-healthy site drain inside one tick, so the stalled signal
/// only fires on a genuine non-drain.
/// </summary>
public int MaxPagesPerTick { get; set; } = 50;
/// <summary>
/// Number of consecutive non-draining cycles (events returned AND
/// <c>MoreAvailable=true</c>) that must accumulate for a site before the actor
@@ -24,6 +24,10 @@ public sealed class SiteAuditReconciliationOptionsValidator
$"AuditLog:Reconciliation:{nameof(SiteAuditReconciliationOptions.BatchSize)} " +
$"({options.BatchSize}) must be 1..2000.");
builder.RequireThat(options.MaxPagesPerTick is > 0 and <= 1000,
$"AuditLog:Reconciliation:{nameof(SiteAuditReconciliationOptions.MaxPagesPerTick)} " +
$"({options.MaxPagesPerTick}) must be 1..1000.");
builder.RequireThat(options.StalledAfterNonDrainingCycles > 0,
$"AuditLog:Reconciliation:{nameof(SiteAuditReconciliationOptions.StalledAfterNonDrainingCycles)} " +
$"({options.StalledAfterNonDrainingCycles}) must be > 0.");
@@ -0,0 +1,41 @@
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <summary>
/// Fits a site-supplied <see cref="SiteCall"/> to the central <c>SiteCalls</c> column widths before
/// it is written (review 2026-10-21 follow-up). The site's tracking store does not bound
/// <see cref="SiteCall.LastError"/>, while the central column is <c>nvarchar(1024)</c>
/// (<c>SiteCallEntityTypeConfiguration</c>): an over-width error used to fail the
/// cached-telemetry dual-write with SQL 8152 — the audit row was reported Rejected to the site
/// and the reconciliation pull's upsert failed the same way, so the <c>SiteCalls</c> row was never
/// written at all. Truncating a diagnostic message is the right trade; dropping the row is not.
/// </summary>
internal static class SiteCallColumnBounds
{
/// <summary>Width of <c>SiteCalls.LastError</c>; must match <c>SiteCallEntityTypeConfiguration</c>.</summary>
public const int LastErrorMaxLength = 1024;
/// <summary>
/// Returns <paramref name="siteCall"/> unchanged when it already fits, otherwise a copy whose
/// <see cref="SiteCall.LastError"/> is truncated to <see cref="LastErrorMaxLength"/> characters
/// (never splitting a UTF-16 surrogate pair).
/// </summary>
/// <param name="siteCall">The row about to be written centrally.</param>
/// <returns>A row that fits the central columns.</returns>
public static SiteCall Clamp(SiteCall siteCall)
{
if (siteCall.LastError is not { Length: > LastErrorMaxLength } error)
{
return siteCall;
}
var cut = LastErrorMaxLength;
if (char.IsHighSurrogate(error[cut - 1]))
{
cut--;
}
return siteCall with { LastError = error[..cut] };
}
}
@@ -150,7 +150,7 @@ public sealed class ScadaBridgeAuditRedactor : IAuditRedactor, IDisposable
}
}
/// <inheritdoc />
/// <summary>Disposes the options-change registration, if any.</summary>
public void Dispose() => _optionsReload?.Dispose();
/// <summary>
@@ -0,0 +1,66 @@
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
/// <summary>
/// Site-only forwarding states that extend <see cref="AuditForwardState"/> inside the site
/// SQLite sidecar (<c>audit_forward_state.ForwardState</c> is TEXT, so these need no schema
/// change). Both mean "left the telemetry PUSH queue without a central ack, still owed to
/// central through the reconciliation PULL": <c>ReadPendingSinceAsync</c> serves them,
/// <c>MarkReconciledUpToAsync</c> retires them, and the retention purge never drops them
/// (review 2026-10-21 06/PERF-1, 06/IMPL-1).
/// </summary>
/// <remarks>
/// They are deliberately NOT members of the Commons <see cref="AuditForwardState"/> enum: they
/// never leave the site store (the wire and the central table carry no forward state), and a
/// downgraded node simply ignores rows in a state it does not know.
/// </remarks>
public static class SiteAuditForwardStates
{
/// <summary>
/// A cached-lifecycle row whose operational (<c>SiteCalls</c>) half can never be built — its
/// tracking snapshot is gone — so the combined-telemetry drain gave up pushing it. Its audit
/// half still reaches central through the reconciliation pull.
/// </summary>
public const string PushAbandoned = "PushAbandoned";
/// <summary>
/// A row central explicitly and permanently refused (listed in the ack's <c>rejected_event_ids</c>) on
/// <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteAuditTelemetryOptions.MaxRejectedPushAttempts"/>
/// consecutive drains. Moved out of the push queue so one poison row cannot wedge every push
/// batch behind it; the pull then delivers it, or central's permanent-abandon path records the
/// loss as a <c>ReconciliationAbandoned</c> row and retires it.
/// </summary>
public const string Rejected = "Rejected";
}
/// <summary>
/// Push-side lifecycle transitions the <c>SiteAuditTelemetryActor</c> needs beyond the Commons
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue"/> surface.
/// Implemented by <see cref="SqliteAuditWriter"/>; the actor discovers it by type-testing the
/// injected queue, so composition roots and test doubles that only implement
/// <c>ISiteAuditQueue</c> keep working (the actor falls back to the pre-review behaviour).
/// </summary>
public interface ISiteAuditPushLifecycle
{
/// <summary>
/// Flips the supplied EventIds from <see cref="AuditForwardState.Pending"/> to
/// <see cref="SiteAuditForwardStates.PushAbandoned"/>. Rows in any other state are untouched.
/// </summary>
/// <param name="eventIds">Event IDs whose push is abandoned.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task MarkPushAbandonedAsync(IReadOnlyList<Guid> eventIds, CancellationToken ct = default);
/// <summary>
/// Records one explicitly-refused push attempt for each supplied <see cref="AuditForwardState.Pending"/>
/// row (bumps <c>AttemptCount</c>, stamps <c>LastAttemptUtc</c>) and flips every row that has
/// now reached <paramref name="maxAttempts"/> to <see cref="SiteAuditForwardStates.Rejected"/>.
/// </summary>
/// <param name="eventIds">Event IDs central listed as permanently rejected.</param>
/// <param name="maxAttempts">Refused attempts after which a row leaves the push queue.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>A task that resolves to the number of rows flipped to Rejected by this call.</returns>
Task<int> RecordRejectedPushAttemptsAsync(
IReadOnlyList<Guid> eventIds, int maxAttempts, CancellationToken ct = default);
}
@@ -0,0 +1,28 @@
using System.Diagnostics.Metrics;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
/// <summary>
/// Site audit forwarding instruments. Published on a meter named
/// <see cref="ScadaBridgeTelemetry.MeterName"/> — OpenTelemetry matches meters by NAME, so these
/// export through the existing <c>ObservedMeters</c> allowlist entry without a Host change.
/// </summary>
internal static class SiteAuditTelemetryMetrics
{
private static readonly Meter Meter = new(ScadaBridgeTelemetry.MeterName);
private static readonly Counter<long> RejectedRows =
Meter.CreateCounter<long>("scadabridge.site.audit.rejected_rows", unit: "1",
description: "Site audit rows central permanently refused MaxRejectedPushAttempts times, moved out of the push queue as Rejected (still served by the reconciliation pull).");
/// <summary>Records <paramref name="count"/> rows flipped to Rejected.</summary>
/// <param name="count">Rows flipped.</param>
public static void RecordRejectedRows(long count)
{
if (count > 0)
{
RejectedRows.Add(count);
}
}
}
@@ -43,7 +43,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
/// stamped by central on ingest, not a site column.
/// </para>
/// </remarks>
public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable, IDisposable
public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLifecycle, IAsyncDisposable, IDisposable
{
// Microsoft.Data.Sqlite reports a generic SQLITE_CONSTRAINT (error code 19)
// on a PRIMARY KEY violation; the extended subcode 1555 (SQLITE_CONSTRAINT_PRIMARYKEY)
@@ -108,6 +108,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
_connection.Open();
InitializeSchema();
LoadMaxServedRowId();
// Open a second connection for read-only callers
// (GetBacklogStatsAsync, ReadPendingAsync, ReadPendingSinceAsync,
@@ -134,10 +135,11 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
_writeQueue = Channel.CreateBounded<PendingAuditEvent>(
new BoundedChannelOptions(_options.ChannelCapacity)
{
// The hot-path enqueue must back-pressure if the background
// writer falls behind; a higher-level fallback
// handles truly catastrophic primary failure with a drop-oldest
// ring buffer.
// Wait mode is only what the (never-awaited) async write path would
// do; WriteAsync uses TryWrite exclusively and fails fast with
// SqliteAuditWriterSaturatedException when the queue is full, so no
// caller ever parks on room (review 2026-10-21 06/MEM-1). The
// FallbackAuditWriter's drop-oldest ring absorbs the overflow.
FullMode = BoundedChannelFullMode.Wait,
SingleReader = true,
SingleWriter = false,
@@ -174,7 +176,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
pragmaCmd.ExecuteNonQuery();
}
// synchronous=NORMAL (arch-review remediation WP1.2): SQLite's FULL default
// synchronous=NORMAL: SQLite's FULL default
// fsyncs on every transaction commit, which is the dominant cost of the
// per-event write path this same change coalesces via FlushIntervalMs — FULL
// would still fsync once per FLUSH even after that fix. NORMAL fsyncs at
@@ -266,6 +268,16 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
CREATE INDEX IF NOT EXISTS IX_fwd
ON audit_forward_state (ForwardState, IsCachedKind, OccurredAtUtc);
-- Node-local key/value state that must survive a process restart. Holds the
-- reconciliation pull's served-rows high-water mark ('max_served_rowid'), so a
-- restart no longer resets the bound that lets a covering cursor retire rows the
-- pull already delivered (review 2026-10-21 06/PERF-1 adversarial follow-up).
-- Additive and idempotent: an older binary simply ignores the table.
CREATE TABLE IF NOT EXISTS audit_site_state (
Key TEXT NOT NULL PRIMARY KEY,
Value INTEGER NOT NULL
);
-- Flow Debugger (design 2026-09-05 §5.4): PullTraceEvents reads audit_event by
-- time window + json_extract on ExecutionId/ParentExecutionId; this bounds the scan
-- to the window. Additive and idempotent.
@@ -286,7 +298,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
/// <summary>
/// Enqueues an audit event for asynchronous batched persistence to SQLite.
/// Back-pressure is applied when the write channel is full.
/// Never parks the caller on queue room: a full queue faults the returned task with
/// <see cref="SqliteAuditWriterSaturatedException"/> immediately.
/// </summary>
/// <param name="evt">The audit event to persist.</param>
/// <param name="ct">Cancellation token.</param>
@@ -300,35 +313,47 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
// alongside the canonical row in the same transaction.
var pending = new PendingAuditEvent(evt, AuditForwardState.Pending);
// CreateBounded(FullMode=Wait) means WriteAsync will await room rather
// than throw when full — exactly the hot-path back-pressure semantics
// we want.
if (!_writeQueue.Writer.TryWrite(pending))
if (_writeQueue.Writer.TryWrite(pending))
{
// The writer is either completed (logger disposed) or the channel
// is at capacity. Fall back to the async path which honours the
// FullMode=Wait policy.
return WriteSlowPathAsync(pending, ct);
return pending.Completion.Task;
}
return pending.Completion.Task;
// TryWrite fails for exactly two reasons: the writer is completed (disposed)
// or the queue is at capacity. The second used to fall into an awaited
// WriteAsync on the channel — one parked waiter, holding its AuditEvent, per
// fire-and-continue script-thread caller for as long as the background writer
// stayed stuck (e.g. behind a multi-minute retention purge). That count was
// unbounded and invisible. Fail fast instead: FallbackAuditWriter routes the
// event to its bounded drop-oldest ring and bumps SiteAuditWriteFailures, which
// is the documented best-effort degradation (review 2026-10-21 06/MEM-1).
if (Volatile.Read(ref _writerCompleted))
{
return Task.FromException(new ObjectDisposedException(nameof(SqliteAuditWriter),
"Event could not be recorded: the audit writer has been disposed."));
}
Interlocked.Increment(ref _saturatedRejections);
return Task.FromException(new SqliteAuditWriterSaturatedException(_options.ChannelCapacity));
}
private async Task WriteSlowPathAsync(PendingAuditEvent pending, CancellationToken ct)
{
try
{
await _writeQueue.Writer.WriteAsync(pending, ct).ConfigureAwait(false);
}
catch (ChannelClosedException)
{
pending.Completion.TrySetException(
new ObjectDisposedException(nameof(SqliteAuditWriter),
"Event could not be recorded: the audit writer has been disposed."));
}
/// <summary>
/// Number of <see cref="WriteAsync"/> calls refused because the write queue was full
/// since construction. Observability seam; the health-report count is the
/// <c>SiteAuditWriteFailures</c> bump <see cref="FallbackAuditWriter"/> makes per refusal.
/// </summary>
public long SaturatedRejectionCount => Interlocked.Read(ref _saturatedRejections);
private long _saturatedRejections;
await pending.Completion.Task.ConfigureAwait(false);
}
// Set (before the channel is completed) once DisposeAsync starts, so a refused
// TryWrite can tell "disposed" from "full" without racing the channel's state.
private bool _writerCompleted;
/// <summary>
/// Rows moved to <see cref="SiteAuditForwardStates.Rejected"/> since construction (also exported
/// as the <c>scadabridge.site.audit.rejected_rows</c> counter).
/// </summary>
public long RejectedPushRowCount => Interlocked.Read(ref _rejectedPushRows);
private long _rejectedPushRows;
private async Task ProcessWriteQueueAsync()
{
@@ -346,8 +371,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
// burst of concurrent enqueues. Once the immediately-available
// burst is drained, keep the transaction open for up to
// FlushIntervalMs waiting for MORE events to trickle in — this is
// the soft-flush coalescing arch-review remediation WP1.2 adds:
// without it, a low-rate trickle of writes (one script call every
// the soft-flush coalescing
// this loop provides: without it, a low-rate trickle of writes (one script call every
// few ms, well under BatchSize) fsyncs once PER EVENT, because the
// writer loop reaches an empty channel and flushes immediately.
// FlushIntervalMs bounds the worst-case added latency any single
@@ -392,12 +417,17 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
/// <summary>
/// Total number of <see cref="FlushBatch"/> transactions committed since
/// construction. Test-only observability seam (arch-review remediation
/// WP1.2) for asserting that FlushIntervalMs coalesces a trickle of
/// construction. Test-only observability seam for asserting that FlushIntervalMs coalesces a trickle of
/// near-simultaneous writes into a single transaction rather than one
/// commit per event; production code never reads this.
/// </summary>
internal int FlushCountForTests => _flushCountForTests;
/// <summary>
/// Test-only: the write lock FlushBatch / the flips / the purge serialise on, so a test can
/// park the background writer deterministically (e.g. to fill the write queue).
/// </summary>
internal object WriteLockForTests => _writeLock;
private int _flushCountForTests;
private void FlushBatch(IReadOnlyList<PendingAuditEvent> batch)
@@ -499,8 +529,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
{
eventCmd.ExecuteNonQuery();
fwdCmd.ExecuteNonQuery();
// Do NOT complete pending.Completion here (arch-review 2026-08-19
// P4-2). A row's INSERTs succeeding is not durable until the
// Do NOT complete pending.Completion here. A row's INSERTs succeeding is not durable until the
// transaction actually commits below — completing the caller's Task
// this early means a later commit failure (SQLITE_FULL, IOERR, or a
// later row in this same batch throwing) reports false success: the
@@ -790,9 +819,11 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
/// <summary>
/// Returns up to <paramref name="batchSize"/> pending or forwarded audit events
/// with <see cref="AuditEvent.OccurredAtUtc"/> &gt;= <paramref name="sinceUtc"/>, oldest first.
/// Used by the reconciliation-pull handler.
/// Returns up to <paramref name="batchSize"/> audit events central has not yet acknowledged
/// (<see cref="AuditForwardState.Pending"/>, <see cref="SiteAuditForwardStates.PushAbandoned"/>,
/// <see cref="SiteAuditForwardStates.Rejected"/>) with <see cref="AuditEvent.OccurredAtUtc"/>
/// &gt;= <paramref name="sinceUtc"/>, oldest first. <see cref="AuditForwardState.Forwarded"/> rows
/// are NOT served (review 2026-10-21 06/PERF-1). Used by the reconciliation-pull handler.
/// </summary>
/// <param name="sinceUtc">Lower bound timestamp (UTC) for event occurrence.</param>
/// <param name="batchSize">Maximum number of rows to return.</param>
@@ -816,9 +847,16 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
// Read via _readConnection / _readLock — same lock-
// decoupling as ReadPendingAsync. JOIN the sidecar; the range scan
// is on the sidecar's duplicated OccurredAtUtc so it stays on IX_fwd.
// Both Pending and Forwarded rows are returned (the central reconciliation
// puller dedups on EventId; re-shipping a Forwarded-but-not-yet-ingested
// row is safe).
//
// ONLY rows central has NOT acknowledged are served: Pending, plus the
// push-exit states PushAbandoned / Rejected (review 2026-10-21 06/PERF-1).
// Forwarded rows used to be served too, which made the pull re-ship every
// row the push had already delivered — central re-inserted each one, and the
// pull's 256-rows-per-tick budget was spent on acked rows, so any site above
// ~0.85 audit rows/s fell permanently behind and latched the stalled signal.
// A Forwarded row is acked by central; the pull has nothing to repair there.
IReadOnlyList<AuditEvent> rows;
long batchMaxRowId;
lock (_readLock)
{
ObjectDisposedException.ThrowIf(_disposed, this);
@@ -840,13 +878,12 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
fs.rowid
FROM audit_event ae
JOIN audit_forward_state fs ON fs.EventId = ae.EventId
WHERE fs.ForwardState IN ($pending, $forwarded)
WHERE fs.ForwardState IN ($pending, $pushAbandoned, $rejected)
AND {cursorPredicate}
ORDER BY fs.OccurredAtUtc ASC, ae.EventId ASC
LIMIT $limit;
""";
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
cmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString());
AddOwedStateParameters(cmd);
// Normalise to UTC ISO-8601 round-trip format to match how OccurredAtUtc
// is stored on insert ("o" format) — string comparison is monotonic for
// that encoding so we can index-scan against it.
@@ -859,14 +896,208 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
cmd.Parameters.AddWithValue("$limit", batchSize);
return Task.FromResult(ReadServedRows(cmd, batchSize));
rows = ReadServedRows(cmd, batchSize, out batchMaxRowId);
}
// Advance (and persist) the mark OUTSIDE _readLock so the two locks are never nested in
// read→write order, and UNDER _writeLock — the lock the purge's rowid clamp runs under.
// Advancing it under _readLock from a value sampled before the SELECT let a purge clamp
// that landed in between be overwritten with the stale-high value (review 2026-10-21
// follow-up L1). Taking max(current, batch max) under _writeLock sees any clamp that
// already happened; the served rows themselves are owed and so can never be purged.
// Only when it moved — at most one tiny upsert per pull.
lock (_writeLock)
{
if (!_disposed && batchMaxRowId > Volatile.Read(ref _maxServedRowId))
{
Volatile.Write(ref _maxServedRowId, batchMaxRowId);
try
{
PersistMaxServedRowId(batchMaxRowId);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"SqliteAuditWriter could not persist the served high-water mark {Mark}; a restart re-learns it from the next pull.",
batchMaxRowId);
}
}
}
return Task.FromResult(rows);
}
/// <summary>
/// Binds the three "owed to central" states — <c>$pending</c>, <c>$pushAbandoned</c>,
/// <c>$rejected</c> — that the reconciliation pull serves and retires.
/// </summary>
private static void AddOwedStateParameters(SqliteCommand cmd)
{
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
cmd.Parameters.AddWithValue("$pushAbandoned", SiteAuditForwardStates.PushAbandoned);
cmd.Parameters.AddWithValue("$rejected", SiteAuditForwardStates.Rejected);
}
/// <inheritdoc />
public Task MarkPushAbandonedAsync(IReadOnlyList<Guid> eventIds, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(eventIds);
if (eventIds.Count == 0)
{
return Task.CompletedTask;
}
lock (_writeLock)
{
ObjectDisposedException.ThrowIf(_disposed, this);
// Pending → PushAbandoned only. Unlike the old "mark Forwarded" this is honest:
// central never acked these rows, so they stay owed to the reconciliation pull
// (served, retired by a covering cursor, never age-purged) — review 2026-10-21
// 06/PERF-1. Attempt bookkeeping is bumped like any drain pass that touched them.
var now = DateTime.UtcNow.ToString("o", System.Globalization.CultureInfo.InvariantCulture);
using var transaction = _connection.BeginTransaction();
for (int start = 0; start < eventIds.Count; start += MarkForwardStateChunkSize)
{
var chunk = Math.Min(MarkForwardStateChunkSize, eventIds.Count - start);
using var cmd = _connection.CreateCommand();
cmd.Transaction = transaction;
var sb = new System.Text.StringBuilder();
sb.Append("UPDATE audit_forward_state SET ForwardState = $abandoned, ")
.Append("AttemptCount = AttemptCount + 1, LastAttemptUtc = $now ")
.Append("WHERE ForwardState = $pending AND EventId IN (");
AppendIdParameters(cmd, sb, eventIds, start, chunk);
sb.Append(");");
cmd.CommandText = sb.ToString();
cmd.Parameters.AddWithValue("$abandoned", SiteAuditForwardStates.PushAbandoned);
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
cmd.Parameters.AddWithValue("$now", now);
cmd.ExecuteNonQuery();
}
transaction.Commit();
return Task.CompletedTask;
}
}
/// <inheritdoc />
public Task<int> RecordRejectedPushAttemptsAsync(
IReadOnlyList<Guid> eventIds, int maxAttempts, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(eventIds);
if (eventIds.Count == 0)
{
return Task.FromResult(0);
}
var threshold = Math.Max(1, maxAttempts);
lock (_writeLock)
{
ObjectDisposedException.ThrowIf(_disposed, this);
// Review 2026-10-21 06/IMPL-1: AttemptCount/LastAttemptUtc used to move only on
// the Pending→Forwarded flip, so a row central kept refusing had no history and
// no give-up — and since the push reads oldest-first, it sat at the head of every
// batch forever. Bump on every refused attempt, then move rows that reached the
// threshold out of the push queue into Rejected (still owed to the pull).
var now = DateTime.UtcNow.ToString("o", System.Globalization.CultureInfo.InvariantCulture);
var flipped = 0;
using var transaction = _connection.BeginTransaction();
for (int start = 0; start < eventIds.Count; start += MarkForwardStateChunkSize)
{
var chunk = Math.Min(MarkForwardStateChunkSize, eventIds.Count - start);
using (var bump = _connection.CreateCommand())
{
bump.Transaction = transaction;
var sb = new System.Text.StringBuilder();
sb.Append("UPDATE audit_forward_state SET AttemptCount = AttemptCount + 1, LastAttemptUtc = $now ")
.Append("WHERE ForwardState = $pending AND EventId IN (");
AppendIdParameters(bump, sb, eventIds, start, chunk);
sb.Append(");");
bump.CommandText = sb.ToString();
bump.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
bump.Parameters.AddWithValue("$now", now);
bump.ExecuteNonQuery();
}
using (var reject = _connection.CreateCommand())
{
reject.Transaction = transaction;
var sb = new System.Text.StringBuilder();
sb.Append("UPDATE audit_forward_state SET ForwardState = $rejected ")
.Append("WHERE ForwardState = $pending AND AttemptCount >= $max AND EventId IN (");
AppendIdParameters(reject, sb, eventIds, start, chunk);
sb.Append(");");
reject.CommandText = sb.ToString();
reject.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
reject.Parameters.AddWithValue("$rejected", SiteAuditForwardStates.Rejected);
reject.Parameters.AddWithValue("$max", threshold);
flipped += reject.ExecuteNonQuery();
}
}
transaction.Commit();
if (flipped > 0)
{
_logger.LogWarning(
"{Count} site audit row(s) were refused by central on {Max} consecutive pushes and moved out " +
"of the push queue as Rejected; the reconciliation pull still serves them (central records " +
"a ReconciliationAbandoned row if they can never be inserted).",
flipped, threshold);
// Its OWN counter, not SiteAuditWriteFailures: the local write succeeded, and
// the refusal is central's verdict on the row — blaming the site's writer for it
// would be wrong (review 2026-10-21 adversarial follow-up).
Interlocked.Add(ref _rejectedPushRows, flipped);
SiteAuditTelemetryMetrics.RecordRejectedRows(flipped);
}
return Task.FromResult(flipped);
}
}
private static void AppendIdParameters(
SqliteCommand cmd, System.Text.StringBuilder sb, IReadOnlyList<Guid> eventIds, int start, int chunk)
{
for (int i = 0; i < chunk; i++)
{
if (i > 0) sb.Append(',');
var p = $"$id{i}";
sb.Append(p);
cmd.Parameters.AddWithValue(p, eventIds[start + i].ToString());
}
}
/// <summary>
/// Test/diagnostic read of one row's sidecar state: <c>(ForwardState, AttemptCount)</c>, or
/// null when the row does not exist.
/// </summary>
/// <param name="eventId">The event to look up.</param>
/// <returns>The row's forward state and attempt count, or null.</returns>
internal (string State, int AttemptCount)? GetForwardStateForTests(Guid eventId)
{
lock (_readLock)
{
ObjectDisposedException.ThrowIf(_disposed, this);
using var cmd = _readConnection.CreateCommand();
cmd.CommandText = "SELECT ForwardState, AttemptCount FROM audit_forward_state WHERE EventId = $id;";
cmd.Parameters.AddWithValue("$id", eventId.ToString());
using var reader = cmd.ExecuteReader();
return reader.Read() ? (reader.GetString(0), reader.GetInt32(1)) : null;
}
}
/// <summary>Flow Debugger PullTraceEvents: hard cap on ids per read (design §5.4).</summary>
public const int MaxExecutionIdsPerRead = 64;
/// <inheritdoc />
/// <summary>
/// Reads audit events belonging to any of the given execution ids, newer than <paramref name="sinceUtc"/>, in keyset-paged order.
/// </summary>
/// <param name="executionIds">Execution ids to match (capped at <see cref="MaxExecutionIdsPerRead"/>).</param>
/// <param name="sinceUtc">Lower bound on the event's occurrence time.</param>
/// <param name="limit">Maximum number of events to return; must be positive.</param>
/// <param name="ct">Cancellation token.</param>
/// <param name="afterEventId">Optional keyset cursor; only events after this id are returned.</param>
/// <returns>The matching audit events.</returns>
public Task<IReadOnlyList<AuditEvent>> ReadByExecutionIdsAsync(
IReadOnlyCollection<Guid> executionIds, DateTime sinceUtc, int limit, CancellationToken ct = default, string? afterEventId = null)
{
@@ -925,6 +1156,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
/// UNION deduplicates rows matching both IDs. Keeping payloads outside the union bounds sorting
/// to IDs/timestamps and avoids the optimizer preferring a whole-store timestamp scan.
/// </summary>
/// <param name="inList">The pre-built SQL parameter list for the <c>IN (...)</c> clauses.</param>
/// <returns>The trace-read SQL text.</returns>
internal static string BuildTraceReadSql(string inList) => $"""
SELECT e.EventId, e.OccurredAtUtc, e.Actor, e.Action, e.Outcome,
e.Category, e.Target, e.SourceNode, e.CorrelationId, e.DetailsJson
@@ -953,23 +1186,68 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
/// rows central could possibly have received through the pull path. Bounds
/// <see cref="MarkReconciledUpToAsync"/> so a row inserted after that point can never be
/// retired by a cursor that happens to sit above its timestamp. Monotonic (a later, smaller
/// batch never lowers it) except for the purge clamp. Written under <c>_readLock</c> (the
/// read path) and read under <c>_writeLock</c> (the flip/purge paths) — two different
/// locks, so the accesses go through <see cref="Volatile"/> for visibility. A stale read
/// can only make the bound smaller, i.e. more conservative, never unsafe.
/// batch never lowers it) except for the purge clamp. Every WRITE happens under
/// <c>_writeLock</c> (the pull's advance and the purge's clamp alike — review 2026-10-21
/// follow-up L1), so a clamp can never be overwritten with a stale-high value; reads from
/// other paths go through <see cref="Volatile"/> for visibility. A stale read can only make
/// the bound smaller, i.e. more conservative, never unsafe.
/// </summary>
private long _maxServedRowId;
private const string MaxServedRowIdKey = "max_served_rowid";
/// <summary>
/// Restores <see cref="_maxServedRowId"/> from <c>audit_site_state</c>, clamped to the live
/// <c>MAX(rowid)</c> so a stale value can never vouch for a reused rowid. Before this the
/// bound was per-process: after a site-node restart it was 0, and because a healthy site's
/// pull now often serves nothing, owed rows central had already ingested (e.g. PushAbandoned)
/// could stay unretired — and counted on the backlog — indefinitely.
/// </summary>
private void LoadMaxServedRowId()
{
using var cmd = _connection.CreateCommand();
cmd.CommandText = """
SELECT MIN(
IFNULL((SELECT Value FROM audit_site_state WHERE Key = $key), 0),
IFNULL((SELECT MAX(rowid) FROM audit_forward_state), 0));
""";
cmd.Parameters.AddWithValue("$key", MaxServedRowIdKey);
var value = Convert.ToInt64(cmd.ExecuteScalar(), System.Globalization.CultureInfo.InvariantCulture);
Volatile.Write(ref _maxServedRowId, value);
}
/// <summary>
/// Durably records the served high-water mark. Runs on the WRITE connection under
/// <c>_writeLock</c> (never nested inside <c>_readLock</c>). Best-effort: a failure leaves
/// the in-memory bound intact and only means a restart re-learns it from the next pull.
/// </summary>
private void PersistMaxServedRowId(long value, SqliteTransaction? transaction = null)
{
using var cmd = _connection.CreateCommand();
cmd.Transaction = transaction;
cmd.CommandText = """
INSERT INTO audit_site_state (Key, Value) VALUES ($key, $value)
ON CONFLICT(Key) DO UPDATE SET Value = excluded.Value;
""";
cmd.Parameters.AddWithValue("$key", MaxServedRowIdKey);
cmd.Parameters.AddWithValue("$value", value);
cmd.ExecuteNonQuery();
}
/// <summary>Test seam: the current in-memory served high-water mark.</summary>
internal long MaxServedRowIdForTests => Volatile.Read(ref _maxServedRowId);
/// <summary>
/// Executes a reconciliation-pull read whose projection carries <c>fs.rowid</c> as an
/// 11th column, materialising the canonical rows while advancing
/// <see cref="_maxServedRowId"/>. Serving a row is what makes it eligible for
/// cursor-proved retirement, so the two must happen in the same step.
/// 11th column, materialising the canonical rows and reporting the highest rowid served.
/// The caller folds that into <see cref="_maxServedRowId"/> under <c>_writeLock</c> before
/// returning the rows — serving a row is what makes it eligible for cursor-proved
/// retirement, so the mark must cover a batch before central can see it.
/// </summary>
private IReadOnlyList<AuditEvent> ReadServedRows(SqliteCommand cmd, int capacityHint)
private IReadOnlyList<AuditEvent> ReadServedRows(SqliteCommand cmd, int capacityHint, out long maxRowId)
{
var rows = new List<AuditEvent>(Math.Min(capacityHint, 256));
var maxRowId = Volatile.Read(ref _maxServedRowId);
maxRowId = 0;
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
@@ -980,7 +1258,6 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
}
Volatile.Write(ref _maxServedRowId, maxRowId);
return rows;
}
@@ -1021,27 +1298,29 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
//
// fs.rowid is insertion order, so bounding the flip at the highest rowid this
// instance has ever served restores the invariant exactly: rows inserted after
// that point are out of the flip's reach no matter where the cursor sits. A
// FORWARDED row is exempt from the bound — central ACKED it through the
// telemetry push path, which is receipt-proof independent of the pull.
// that point are out of the flip's reach no matter where the cursor sits.
//
// Bound state is per-process: after a restart it is 0 until this instance serves
// a batch, so the first pull retires only Forwarded rows and the pull after it
// resumes normal retirement. Conservative in the safe direction (delays
// retirement, never loses a row); the liveness note in MarkReconciledUpToAsync's
// interface doc + Component-AuditLog.md covers what a permanently halted
// reconciliation means for such rows.
// Only the OWED states flip (Pending / PushAbandoned / Rejected — exactly what
// ReadPendingSinceAsync serves). Forwarded rows are no longer touched: central
// acked them over the push path, they are already purge-eligible, and flipping
// them was one UPDATE per pushed row for no benefit (review 2026-10-21 06/PERF-1).
//
// The bound is persisted in audit_site_state (loaded at construction, clamped to
// the live MAX(rowid)), so a site-node restart no longer resets it to 0 and strands
// rows the pull had already served. Conservative in the safe direction (it can
// only delay retirement, never lose a row); the liveness note in
// MarkReconciledUpToAsync's interface doc + Component-AuditLog.md covers what a
// permanently halted reconciliation means for such rows.
using var cmd = _connection.CreateCommand();
cmd.CommandText = $"""
UPDATE audit_forward_state AS fs
SET ForwardState = $reconciled
WHERE fs.ForwardState IN ($pending, $forwarded)
AND (fs.ForwardState = $forwarded OR fs.rowid <= $maxServedRowId)
WHERE fs.ForwardState IN ($pending, $pushAbandoned, $rejected)
AND fs.rowid <= $maxServedRowId
AND {predicate};
""";
cmd.Parameters.AddWithValue("$reconciled", AuditForwardState.Reconciled.ToString());
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
cmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString());
AddOwedStateParameters(cmd);
cmd.Parameters.AddWithValue("$maxServedRowId", Volatile.Read(ref _maxServedRowId));
cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString(
"o", System.Globalization.CultureInfo.InvariantCulture));
@@ -1067,8 +1346,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
{
ObjectDisposedException.ThrowIf(_disposed, this);
// Flip the sidecar from Pending/Forwarded → Reconciled. Rows
// already Reconciled are left untouched (idempotent re-call), and the
// Flip the sidecar from Pending/Forwarded (or a push-exit state) → Reconciled.
// Rows already Reconciled are left untouched (idempotent re-call), and the
// canonical audit_event row is never modified.
var reconciled = AuditForwardState.Reconciled.ToString();
var pending = AuditForwardState.Pending.ToString();
@@ -1084,7 +1363,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
cmd.Transaction = transaction;
var sb = new System.Text.StringBuilder();
sb.Append("UPDATE audit_forward_state SET ForwardState = $reconciled ")
.Append("WHERE ForwardState IN ($pending, $forwarded) AND EventId IN (");
.Append("WHERE ForwardState IN ($pending, $forwarded, $pushAbandoned, $rejected) AND EventId IN (");
for (int i = 0; i < chunk; i++)
{
if (i > 0) sb.Append(',');
@@ -1097,6 +1376,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
cmd.Parameters.AddWithValue("$reconciled", reconciled);
cmd.Parameters.AddWithValue("$pending", pending);
cmd.Parameters.AddWithValue("$forwarded", forwarded);
cmd.Parameters.AddWithValue("$pushAbandoned", SiteAuditForwardStates.PushAbandoned);
cmd.Parameters.AddWithValue("$rejected", SiteAuditForwardStates.Rejected);
cmd.ExecuteNonQuery();
}
transaction.Commit();
@@ -1124,12 +1405,15 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
ObjectDisposedException.ThrowIf(_disposed, this);
using var cmd = _readConnection.CreateCommand();
// "Pending" on the health report means OWED to central: the push-exit states
// (PushAbandoned / Rejected) still await the reconciliation pull and are never
// purged, so they count toward the backlog and its oldest-row age.
cmd.CommandText = """
SELECT COUNT(*), MIN(OccurredAtUtc)
FROM audit_forward_state
WHERE ForwardState = $pending;
WHERE ForwardState IN ($pending, $pushAbandoned, $rejected);
""";
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
AddOwedStateParameters(cmd);
using var reader = cmd.ExecuteReader();
reader.Read();
@@ -1173,7 +1457,52 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
/// <inheritdoc />
public Task<int> PurgeExpiredAsync(DateTime olderThanUtc, CancellationToken ct = default)
/// <remarks>
/// <b>Paged (review 2026-10-21 06/MEM-1).</b> The purge deletes at most
/// <see cref="SqliteAuditWriterOptions.PurgeBatchSize"/> rows per transaction and runs a
/// bounded <c>PRAGMA incremental_vacuum(N)</c> after each page, taking <c>_writeLock</c>
/// per page and releasing it in between, so the batched hot-path writer interleaves with
/// a multi-million-row purge rather than stalling behind one giant DELETE + an unbounded
/// vacuum. Every eligible row is still removed by one call; only the lock grain changed.
/// </remarks>
public async Task<int> PurgeExpiredAsync(DateTime olderThanUtc, CancellationToken ct = default)
{
// Cutoff uses the same round-trip "o" encoding the writer stores
// OccurredAtUtc in, so string comparison stays monotonic (see FlushBatch).
var cutoff = EnsureUtc(olderThanUtc).ToString(
"o", System.Globalization.CultureInfo.InvariantCulture);
var pageSize = Math.Max(1, _options.PurgeBatchSize);
var total = 0;
while (true)
{
ct.ThrowIfCancellationRequested();
var purged = PurgePage(cutoff, pageSize);
total += purged;
if (purged < pageSize)
{
return total;
}
// Lock already released by PurgePage; yield so the background writer (and any
// flip/read waiting on _writeLock) gets the lock before the next page.
await Task.Yield();
}
}
/// <summary>
/// Number of purge-page transactions committed since construction. Test-only
/// observability seam for asserting the purge is paged; production code never reads it.
/// </summary>
internal int PurgePageCountForTests => Volatile.Read(ref _purgePageCountForTests);
private int _purgePageCountForTests;
/// <summary>
/// Deletes ONE page (at most <paramref name="pageSize"/> rows) of purge-eligible rows in a
/// single transaction under <c>_writeLock</c>, then reclaims a bounded number of freed
/// pages. Returns the number of canonical rows deleted.
/// </summary>
private int PurgePage(string cutoff, int pageSize)
{
// Runs on the WRITE connection under _writeLock (a DELETE mutates the file),
// mirroring MarkForwardedAsync / MarkReconciledAsync. Serialising with the
@@ -1182,24 +1511,18 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
{
ObjectDisposedException.ThrowIf(_disposed, this);
// Cutoff uses the same round-trip "o" encoding the writer stores
// OccurredAtUtc in, so string comparison stays monotonic (see FlushBatch).
var cutoff = EnsureUtc(olderThanUtc).ToString(
"o", System.Globalization.CultureInfo.InvariantCulture);
int purged;
using (var transaction = _connection.BeginTransaction())
{
try
{
// Capture the eligible EventIds from the sidecar FIRST: the
// Forwarded/Reconciled + age criteria live on
// audit_forward_state, and it is the FK child of audit_event, so
// it must be deleted before the parent. A temp table (rather than
// an IN (...) list) keeps an arbitrarily large purge set off the
// SQLite 999-parameter limit. The hard invariant is encoded here:
// only Forwarded/Reconciled rows qualify — Pending is never purged
// on age alone.
// Capture this page's eligible EventIds from the sidecar FIRST: the
// Forwarded/Reconciled + age criteria live on audit_forward_state, and
// it is the FK child of audit_event, so it must be deleted before the
// parent. A temp table (rather than an IN (...) list) keeps the page off
// the SQLite 999-parameter limit. The hard invariant is encoded here:
// only Forwarded/Reconciled rows qualify — Pending (and the pull-owed
// PushAbandoned/Rejected states) are never purged on age alone.
using (var createCmd = _connection.CreateCommand())
{
createCmd.Transaction = transaction;
@@ -1208,11 +1531,13 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
CREATE TEMP TABLE purge_ids AS
SELECT EventId FROM audit_forward_state
WHERE ForwardState IN ($forwarded, $reconciled)
AND OccurredAtUtc < $cutoff;
AND OccurredAtUtc < $cutoff
LIMIT $limit;
""";
createCmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString());
createCmd.Parameters.AddWithValue("$reconciled", AuditForwardState.Reconciled.ToString());
createCmd.Parameters.AddWithValue("$cutoff", cutoff);
createCmd.Parameters.AddWithValue("$limit", pageSize);
createCmd.ExecuteNonQuery();
}
@@ -1264,10 +1589,14 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
var liveMax = Convert.ToInt64(maxCmd.ExecuteScalar(),
System.Globalization.CultureInfo.InvariantCulture);
if (liveMax < Volatile.Read(ref _maxServedRowId))
{
Volatile.Write(ref _maxServedRowId, liveMax);
PersistMaxServedRowId(liveMax, transaction);
}
}
transaction.Commit();
Interlocked.Increment(ref _purgePageCountForTests);
}
catch
{
@@ -1276,16 +1605,19 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
}
// Reclaim the freed pages back to the OS. auto_vacuum=INCREMENTAL (set in
// InitializeSchema) makes this a bounded incremental step rather than a
// full VACUUM rewrite. Runs outside the transaction; a vacuum failure must
// not fault the purge itself, so it is best-effort.
// Reclaim a BOUNDED number of freed pages back to the OS. A bare
// `PRAGMA incremental_vacuum;` empties the whole freelist in one step, which
// after a large purge held _writeLock for as long as the delete itself did.
// auto_vacuum=INCREMENTAL (set in InitializeSchema) makes this a cheap
// freelist trim. Runs outside the transaction; a vacuum failure must not fault
// the purge itself, so it is best-effort.
if (purged > 0)
{
try
{
using var vacuumCmd = _connection.CreateCommand();
vacuumCmd.CommandText = "PRAGMA incremental_vacuum;";
vacuumCmd.CommandText = string.Create(System.Globalization.CultureInfo.InvariantCulture,
$"PRAGMA incremental_vacuum({Math.Max(1, _options.PurgeVacuumPagesPerBatch)});");
vacuumCmd.ExecuteNonQuery();
}
catch (Exception ex)
@@ -1295,7 +1627,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
}
}
return Task.FromResult(purged);
return purged;
}
}
@@ -1379,6 +1711,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
lock (_writeLock)
{
if (_disposed) return;
Volatile.Write(ref _writerCompleted, true);
// Stop accepting new events. Completing the channel writer is the
// shutdown signal: WriteAsync calls observe the completion and
// fault, and the writer loop drains any already-buffered items
@@ -29,8 +29,11 @@ public sealed class SqliteAuditWriterOptions
/// <summary>
/// Capacity of the bounded write queue. Set high enough that ordinary
/// script bursts never fill it; <see cref="System.Threading.Channels.BoundedChannelFullMode.Wait"/>
/// applies when the writer falls behind.
/// script bursts never fill it. When the writer falls behind and the queue is
/// full, <c>WriteAsync</c> fails fast with <see cref="SqliteAuditWriterSaturatedException"/>
/// instead of parking an unbounded number of waiters (review 2026-10-21 06/MEM-1);
/// <see cref="FallbackAuditWriter"/> turns that into its documented best-effort
/// drop-oldest ring + <c>SiteAuditWriteFailures</c> counter bump.
/// </summary>
public int ChannelCapacity { get; set; } = 4096;
@@ -40,6 +43,23 @@ public sealed class SqliteAuditWriterOptions
/// <summary>Soft flush interval the writer enforces when fewer than BatchSize events are queued.</summary>
public int FlushIntervalMs { get; set; } = 50;
/// <summary>
/// Maximum rows the retention purge deletes in ONE transaction. The purge loops page by
/// page and releases the write lock between pages, so the batched hot-path writer
/// interleaves with a large purge instead of stalling behind one multi-minute DELETE
/// (review 2026-10-21 06/MEM-1 — the same hazard class register row 34 fixed for
/// <c>site_events</c>).
/// </summary>
public int PurgeBatchSize { get; set; } = 5000;
/// <summary>
/// Free pages reclaimed by the bounded <c>PRAGMA incremental_vacuum(N)</c> run after each
/// purge page. Bounds the time the vacuum holds the write lock; pages not reclaimed in
/// one step stay on the freelist and are reused by later inserts or reclaimed by the
/// next purge.
/// </summary>
public int PurgeVacuumPagesPerBatch { get; set; } = 1000;
/// <summary>
/// Poll cadence, in seconds, for the <c>SiteAuditBacklogReporter</c> hosted service
/// that probes the SQLite audit backlog and pushes the snapshot onto the site health
@@ -12,7 +12,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
/// </summary>
/// <remarks>
/// <see cref="SqliteAuditWriterOptions.DatabasePath"/> is deliberately NOT
/// validated here (arch-review remediation WP1.2). <c>AddAuditLog</c> runs on
/// validated here. <c>AddAuditLog</c> runs on
/// BOTH Central and Site composition roots and binds this options type with
/// <c>ValidateOnStart</c> either way, but only Site nodes ever resolve
/// <see cref="SqliteAuditWriter"/> — requiring a non-empty path in this
@@ -37,6 +37,14 @@ public sealed class SqliteAuditWriterOptionsValidator : OptionsValidatorBase<Sql
$"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.FlushIntervalMs)} " +
$"({options.FlushIntervalMs}) must be > 0.");
builder.RequireThat(options.PurgeBatchSize > 0,
$"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.PurgeBatchSize)} " +
$"({options.PurgeBatchSize}) must be > 0.");
builder.RequireThat(options.PurgeVacuumPagesPerBatch > 0,
$"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.PurgeVacuumPagesPerBatch)} " +
$"({options.PurgeVacuumPagesPerBatch}) must be > 0.");
// BacklogPollIntervalSeconds is deliberately unchecked: a non-positive
// value falls back to the reporter's 30s default (see the option's
// remarks + register row 21 resolution).
@@ -0,0 +1,24 @@
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
/// <summary>
/// Thrown (as a faulted task) by <see cref="SqliteAuditWriter.WriteAsync"/> when the bounded
/// write queue is full because the background writer has fallen behind — e.g. it is parked
/// behind a long retention-purge page. Failing fast instead of awaiting room keeps the number
/// of parked callers (each holding its <c>AuditEvent</c>) bounded; <see cref="FallbackAuditWriter"/>
/// catches it like any other primary failure, stashes the event in its drop-oldest ring and
/// bumps the <c>SiteAuditWriteFailures</c> health counter (review 2026-10-21 06/MEM-1).
/// Audit writes are best-effort: this never reaches the user-facing action.
/// </summary>
public sealed class SqliteAuditWriterSaturatedException : InvalidOperationException
{
/// <summary>Initializes a new instance with the queue capacity that was exhausted.</summary>
/// <param name="capacity">The configured <see cref="SqliteAuditWriterOptions.ChannelCapacity"/>.</param>
public SqliteAuditWriterSaturatedException(int capacity)
: base($"Audit write queue is full ({capacity} pending events); the background writer has fallen behind.")
{
Capacity = capacity;
}
/// <summary>The write-queue capacity that was exhausted.</summary>
public int Capacity { get; }
}
@@ -163,8 +163,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
// Route every async dependency call through the
// per-actor lifecycle token so PostStop cancellation actually
// propagates into the queue read, the gRPC push, and the
// mark-forwarded write. OperationCanceledException is swallowed by
// the catch-all below.
// mark-forwarded write. A shutdown-driven OperationCanceledException is
// swallowed silently by the dedicated catch below.
var ct = _lifecycleCts.Token;
try
{
@@ -179,6 +179,7 @@ public class SiteAuditTelemetryActor : ReceiveActor
}
var batch = BuildBatch(pending);
var sentIds = batch.Events.Select(e => e.EventId).ToList();
IngestAck ack;
try
@@ -203,6 +204,14 @@ public class SiteAuditTelemetryActor : ReceiveActor
await _queue.MarkForwardedAsync(acceptedIds, ct)
.ConfigureAwait(false);
}
await RecordRefusalsAsync(sentIds, ack, ct).ConfigureAwait(false);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
// Graceful stop: PostStop cancelled the lifecycle token while the drain
// was awaiting the queue / gRPC push. Expected on every orderly shutdown or
// failover — not an error (review 2026-10-21 06/COR-2).
}
catch (Exception ex)
{
@@ -262,8 +271,11 @@ public class SiteAuditTelemetryActor : ReceiveActor
var batch = new CachedTelemetryBatch();
var wireBytes = AuditWireBudget.InitialBytes;
var emittedEventIds = new List<Guid>(pending.Count);
// EventId → the tracking-row version its packet carried, so an accepted
// packet can be recorded as a central ack (review 2026-10-21 06/PERF-2).
var packetVersions = new Dictionary<Guid, (TrackedOperationId Id, DateTime UpdatedAtUtc)>(pending.Count);
// Rows whose operational half can never be built. They are marked
// Forwarded so they LEAVE this queue — see AbandonUnresolvableAsync
// PushAbandoned so they LEAVE this queue — see AbandonUnresolvableAsync
// for why that is safe and why leaving them Pending is not.
var abandonedEventIds = new List<Guid>();
// Counted, not logged per row: a whole batch can fail identically
@@ -329,6 +341,7 @@ public class SiteAuditTelemetryActor : ReceiveActor
var packet = BuildCachedPacket(auditRow, snapshot);
if (!AuditWireBudget.TryAppend(batch.Packets, packet, ref wireBytes, auditRow.EventId.ToString())) break;
emittedEventIds.Add(auditRow.EventId);
packetVersions[auditRow.EventId] = (snapshot.Id, snapshot.UpdatedAtUtc);
}
if (lookupFailures > 0)
@@ -356,7 +369,7 @@ public class SiteAuditTelemetryActor : ReceiveActor
if (batch.Packets.Count == 0)
{
// Nothing resolvable in this read. Any permanently-unresolvable
// rows have just been marked Forwarded above, so the next drain
// rows have just been marked PushAbandoned above, so the next drain
// sees past them rather than re-reading the same head of queue.
return;
}
@@ -380,7 +393,15 @@ public class SiteAuditTelemetryActor : ReceiveActor
{
await _queue.MarkForwardedAsync(acceptedIds, ct)
.ConfigureAwait(false);
await RecordCentralAcksAsync(acceptedIds, packetVersions, ct).ConfigureAwait(false);
}
await RecordRefusalsAsync(
batch.Packets.Select(p => p.AuditEvent.EventId).ToList(), ack, ct).ConfigureAwait(false);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
// Graceful stop mid-drain — see OnDrainAsync (review 2026-10-21 06/COR-2).
}
catch (Exception ex)
{
@@ -398,33 +419,80 @@ public class SiteAuditTelemetryActor : ReceiveActor
}
/// <summary>
/// Marks cached rows whose operational half can never be built as
/// Forwarded, so they leave the cached-drain queue.
/// Tells the tracking store which tracking-row versions central has just accepted, so
/// the SiteCalls reconciliation pull stops re-serving them (review 2026-10-21
/// 06/PERF-2). Accepted means central committed the packet's <c>SiteCalls</c> upsert
/// in the same transaction as its audit row, i.e. central now holds exactly what a pull
/// of that version would send. Several packets of one operation in a batch collapse to
/// the newest version. Best-effort: the store only enqueues, and a failure here just
/// leaves the rows eligible for one more pull.
/// </summary>
private async Task RecordCentralAcksAsync(
IReadOnlyList<Guid> acceptedIds,
Dictionary<Guid, (TrackedOperationId Id, DateTime UpdatedAtUtc)> packetVersions,
CancellationToken ct)
{
var versions = new Dictionary<TrackedOperationId, DateTime>();
foreach (var eventId in acceptedIds)
{
if (!packetVersions.TryGetValue(eventId, out var v))
{
continue;
}
if (!versions.TryGetValue(v.Id, out var existing) || v.UpdatedAtUtc > existing)
{
versions[v.Id] = v.UpdatedAtUtc;
}
}
if (versions.Count == 0 || _trackingStore is null)
{
return;
}
try
{
await _trackingStore.RecordCentralAcksAsync(versions, ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogDebug(ex,
"Recording {Count} central ack(s) in the tracking store failed; those rows stay eligible for the reconciliation pull.",
versions.Count);
}
}
/// <summary>
/// Moves cached rows whose operational half can never be built out of the
/// cached-drain queue, as <see cref="SiteAuditForwardStates.PushAbandoned"/>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why they must leave.</b> Leaving an unresolvable row Pending — the
/// previous behaviour — is not a harmless skip. The queue is read
/// oldest-first with a fixed <c>BatchSize</c>, so once a batch's worth of
/// unresolvable rows collects at the head, every drain re-reads exactly
/// those rows, fails identically, and never sees the newer rows behind
/// them. The cached-telemetry path stalls permanently, and each pass logs
/// once per row (measured at ~2 800 warnings/minute on a rig).
/// <b>Why they must leave.</b> Leaving an unresolvable row Pending is not a
/// harmless skip. The queue is read oldest-first with a fixed
/// <c>BatchSize</c>, so once a batch's worth of unresolvable rows collects at
/// the head, every drain re-reads exactly those rows, fails identically, and
/// never sees the newer rows behind them. The cached-telemetry path stalls
/// permanently, and each pass logs once per row (measured at ~2 800
/// warnings/minute on a rig).
/// </para>
/// <para>
/// <b>Why Forwarded is the honest state.</b> Its role in this state machine
/// is "no longer owed by the drain, still eligible for reconciliation" —
/// which is exactly the situation here. The audit half is NOT dropped:
/// <c>ReadPendingSinceAsync</c> covers Forwarded rows as well as Pending
/// ones and central dedups on EventId, so the reconciliation pull still
/// delivers them. What is genuinely lost is the operational
/// (<c>SiteCalls</c>) half — and that is unrecoverable regardless, because
/// the tracking row it would have been built from no longer exists.
/// <b>Why PushAbandoned, not Forwarded.</b> These rows were marked Forwarded
/// until review 2026-10-21 (06/PERF-1), which only worked because the
/// reconciliation pull used to re-serve every Forwarded row — the coupling
/// that capped the pull's throughput. Forwarded now means "central acked it"
/// and the pull no longer serves it, so an un-acked row needs a state of its
/// own: <c>PushAbandoned</c> is served by the pull, retired by a covering
/// cursor and never age-purged. The audit half therefore still reaches central;
/// what is genuinely lost is the operational (<c>SiteCalls</c>) half, which is
/// unrecoverable regardless because the tracking row no longer exists.
/// </para>
/// <para>
/// A failure to mark is swallowed: the rows simply stay Pending and the
/// next drain retries. Escalating here would take down the audit drain over
/// a best-effort cleanup.
/// A queue that does not implement <see cref="ISiteAuditPushLifecycle"/> (a
/// test double) falls back to the old Forwarded flip. A failure to mark is
/// swallowed: the rows simply stay Pending and the next drain retries.
/// Escalating here would take down the audit drain over a best-effort cleanup.
/// </para>
/// </remarks>
private async Task AbandonUnresolvableAsync(
@@ -432,24 +500,89 @@ public class SiteAuditTelemetryActor : ReceiveActor
{
try
{
await _queue.MarkForwardedAsync(eventIds, ct).ConfigureAwait(false);
if (_queue is ISiteAuditPushLifecycle lifecycle)
{
await lifecycle.MarkPushAbandonedAsync(eventIds, ct).ConfigureAwait(false);
}
else
{
await _queue.MarkForwardedAsync(eventIds, ct).ConfigureAwait(false);
}
_logger.LogWarning(
"Cached-telemetry drain: abandoned the operational half of {Count} row(s) with no " +
"resolvable tracking snapshot after {Grace}s; marked Forwarded so they no longer block " +
"resolvable tracking snapshot after {Grace}s; marked PushAbandoned so they no longer block " +
"the queue. The audit half still reaches central via the reconciliation pull. This is " +
"expected after tracking retention elapses or the tracking store is reset independently " +
"of the audit store.",
eventIds.Count, graceSeconds);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Cached-telemetry drain: could not mark {Count} unresolvable row(s) Forwarded " +
"Cached-telemetry drain: could not mark {Count} unresolvable row(s) PushAbandoned " +
"(sqlite {SqliteError}); they stay Pending and will be retried.",
eventIds.Count, SqliteErrorCodes.Describe(ex));
}
}
/// <summary>
/// Records a refused push attempt for every SENT row central explicitly listed in the ack's
/// <c>RejectedEventIds</c> — a permanent, row-intrinsic insert failure. Rows refused
/// <see cref="SiteAuditTelemetryOptions.MaxRejectedPushAttempts"/> times leave the push queue as
/// <see cref="SiteAuditForwardStates.Rejected"/> (review 2026-10-21 06/IMPL-1).
/// </summary>
/// <remarks>
/// "Not accepted" is NOT a refusal: central answers with an empty or partial ack during a SQL
/// maintenance window, a timeout, or an exhausted ingest budget, and counting those would flip
/// healthy rows to Rejected. Only the explicit list counts; an older central never sends it, so
/// against it this degrades to the pre-IMPL-1 behaviour (rows stay Pending). Best-effort: a
/// bookkeeping failure leaves the rows Pending.
/// </remarks>
private async Task RecordRefusalsAsync(IReadOnlyList<string> sentIds, IngestAck ack, CancellationToken ct)
{
if (_queue is not ISiteAuditPushLifecycle lifecycle || sentIds.Count == 0 || ack.RejectedEventIds.Count == 0)
{
return;
}
var sent = new HashSet<string>(sentIds, StringComparer.OrdinalIgnoreCase);
var refused = new List<Guid>();
foreach (var raw in ack.RejectedEventIds)
{
if (sent.Contains(raw) && Guid.TryParse(raw, out var id))
{
refused.Add(id);
}
}
if (refused.Count == 0)
{
return;
}
try
{
await lifecycle
.RecordRejectedPushAttemptsAsync(refused, Math.Max(1, _options.MaxRejectedPushAttempts), ct)
.ConfigureAwait(false);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Could not record {Count} refused push attempt(s) (sqlite {SqliteError}); the rows stay Pending.",
refused.Count, SqliteErrorCodes.Describe(ex));
}
}
private static AuditEventBatch BuildBatch(IReadOnlyList<AuditEvent> events)
{
var batch = new AuditEventBatch();
@@ -29,7 +29,7 @@ public sealed class SiteAuditTelemetryOptions
/// <summary>
/// How long a cached-telemetry audit row may go without a resolvable
/// <c>OperationTracking</c> snapshot before the drain ABANDONS its
/// operational half and marks the row Forwarded. Default: 300 s.
/// operational half and marks the row <c>PushAbandoned</c>. Default: 300 s.
/// </summary>
/// <remarks>
/// <para>
@@ -45,8 +45,9 @@ public sealed class SiteAuditTelemetryOptions
/// <para>
/// Abandoning costs only the operational (<c>SiteCalls</c>) half, which is
/// unrecoverable anyway once the tracking row is gone. The audit half is
/// NOT lost: the reconciliation pull covers Forwarded rows as well as
/// Pending ones, and central dedups on EventId.
/// NOT lost: the reconciliation pull serves <c>PushAbandoned</c> rows (central
/// never acked them) exactly as it serves Pending ones, and central dedups on
/// EventId (review 2026-10-21 06/PERF-1).
/// </para>
/// <para>
/// Set to 0 to abandon on the first failed lookup. Negative values are
@@ -54,4 +55,24 @@ public sealed class SiteAuditTelemetryOptions
/// </para>
/// </remarks>
public int CachedTrackingGraceSeconds { get; set; } = 300;
/// <summary>
/// Number of drains in which central EXPLICITLY refused a row (listed it in the ack's
/// additive <c>rejected_event_ids</c> — a permanent, row-intrinsic insert failure such as an
/// over-width value) after which the row leaves the push queue as <c>Rejected</c>.
/// Default: 10 (≈ 50 s at the busy cadence).
/// </summary>
/// <remarks>
/// The push reads oldest-first, so a row central can never insert (e.g. an over-width
/// column) used to sit at the head of every batch forever, failing each set-based insert
/// at central and dragging the batch into the per-row fallback (review 2026-10-21
/// 06/IMPL-1). A transport fault, and a row merely ABSENT from the accepted list (a database
/// outage, a timeout, an exhausted ingest budget), never counts; an older central that does not
/// send the field therefore never triggers it. <c>Rejected</c> is NOT a drop:
/// the reconciliation pull still serves the row, so a transient central refusal still
/// lands, and a genuinely poison row is recorded centrally as <c>ReconciliationAbandoned</c>
/// before it retires. Each flip is counted on <c>scadabridge.site.audit.rejected_rows</c> (NOT
/// <c>SiteAuditWriteFailures</c> — the site's own write succeeded).
/// </remarks>
public int MaxRejectedPushAttempts { get; set; } = 10;
}
@@ -27,6 +27,10 @@ public sealed class SiteAuditTelemetryOptionsValidator : OptionsValidatorBase<Si
$"AuditLog:SiteTelemetry:{nameof(SiteAuditTelemetryOptions.IdleIntervalSeconds)} " +
$"({options.IdleIntervalSeconds}) must be > 0.");
builder.RequireThat(options.MaxRejectedPushAttempts > 0,
$"AuditLog:SiteTelemetry:{nameof(SiteAuditTelemetryOptions.MaxRejectedPushAttempts)} " +
$"({options.MaxRejectedPushAttempts}) must be > 0.");
builder.RequireThat(options.IdleIntervalSeconds >= options.BusyIntervalSeconds,
$"AuditLog:SiteTelemetry:{nameof(SiteAuditTelemetryOptions.IdleIntervalSeconds)} " +
$"({options.IdleIntervalSeconds}) must be >= " +
@@ -82,7 +82,7 @@ public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
.Ask<IngestAuditEventsReply>(new IngestAuditEventsCommand(events), _askTimeout, ct)
.ConfigureAwait(false);
return ToAck(reply.AcceptedEventIds);
return ToAck(reply.AcceptedEventIds, reply.RejectedEventIds);
}
/// <inheritdoc/>
@@ -105,16 +105,21 @@ public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
.Ask<IngestCachedTelemetryReply>(new IngestCachedTelemetryCommand(entries), _askTimeout, ct)
.ConfigureAwait(false);
return ToAck(reply.AcceptedEventIds);
return ToAck(reply.AcceptedEventIds, reply.RejectedEventIds);
}
private static IngestAck ToAck(IReadOnlyList<Guid> acceptedEventIds)
private static IngestAck ToAck(IReadOnlyList<Guid> acceptedEventIds, IReadOnlyList<Guid> rejectedEventIds)
{
var ack = new IngestAck();
foreach (var id in acceptedEventIds)
{
ack.AcceptedEventIds.Add(id.ToString());
}
// Explicit permanent refusals only (review 2026-10-21 06/IMPL-1); empty from an older central.
foreach (var id in rejectedEventIds)
{
ack.RejectedEventIds.Add(id.ToString());
}
return ack;
}
}
@@ -17,6 +17,8 @@ internal static class SqliteErrorCodes
/// "primary/extended" (e.g. "10/522") for a <see cref="SqliteException"/>
/// anywhere in the exception chain; "n/a" for non-SQLite failures.
/// </summary>
/// <param name="ex">The exception to inspect, including its inner exceptions.</param>
/// <returns>The "primary/extended" SQLite error code string, or "n/a" for non-SQLite failures.</returns>
public static string Describe(Exception ex)
{
for (Exception? e = ex; e is not null; e = e.InnerException)
@@ -0,0 +1,62 @@
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
namespace ZB.MOM.WW.ScadaBridge.CLI;
/// <summary>
/// In-memory, process-lifetime store of the bearer tokens the CLI obtains from
/// <c>POST /auth/token</c> (review 2026-10-21 05/U1, 05/P1). One entry per
/// <c>(management URL, username, password)</c> so every <see cref="ManagementHttpClient"/> a
/// single CLI invocation creates shares one credential exchange — one LDAP bind per invocation
/// instead of one per management call.
/// </summary>
/// <remarks>
/// Tokens are never written to disk, logged or echoed; the store dies with the process. The key
/// includes a SHA-256 digest of the password rather than the password itself, so a different
/// password for the same user never reuses a token minted for the old one.
/// </remarks>
internal sealed class CliTokenStore
{
/// <summary>The process-wide store used by the public <see cref="ManagementHttpClient"/> constructor.</summary>
public static CliTokenStore Shared { get; } = new();
private readonly ConcurrentDictionary<string, Session> _sessions = new(StringComparer.Ordinal);
/// <summary>Returns the (shared) session for a management URL and credential pair.</summary>
/// <param name="baseUrl">The management base URL.</param>
/// <param name="username">The CLI username.</param>
/// <param name="password">The CLI password.</param>
/// <returns>The session carrying the cached token, if any.</returns>
public Session For(string baseUrl, string username, string password)
{
var digest = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(password)));
var key = $"{baseUrl.TrimEnd('/').ToLowerInvariant()}\n{username}\n{digest}";
return _sessions.GetOrAdd(key, _ => new Session());
}
/// <summary>Token state for one credential pair against one server.</summary>
internal sealed class Session
{
/// <summary>Serialises token exchanges so concurrent calls share one bind.</summary>
public SemaphoreSlim Gate { get; } = new(1, 1);
/// <summary>The current bearer token, or null when none is held.</summary>
public string? Token { get; set; }
/// <summary>When <see cref="Token"/> should be treated as stale (expiry minus a safety margin).</summary>
public DateTimeOffset RefreshAfter { get; set; }
/// <summary>
/// True once the server has shown it does not take bearer tokens on the management
/// surfaces (no <c>/auth/token</c>, an older server whose tokens nothing accepts, or a
/// login-disabled deployment): every later call goes straight to HTTP Basic.
/// </summary>
public bool BasicOnly { get; set; }
/// <summary>True while a non-stale token is held.</summary>
/// <param name="now">The current instant.</param>
/// <returns>Whether <see cref="Token"/> can be sent.</returns>
public bool HasUsableToken(DateTimeOffset now) => Token is not null && now < RefreshAfter;
}
}
@@ -220,8 +220,8 @@ public static class AuditCommands
// Ctrl-C is the one way to abort an in-flight export other than the
// headers/inactivity deadlines below — mirrors the CancellationTokenSource
// owned by DebugCommands' streaming debug-view command (arch-review
// 2026-08-19 P6-8: this was the one CLI HTTP path with NO deadline at all).
// owned by DebugCommands' streaming debug-view command (this was the
// one CLI HTTP path with NO deadline at all).
using var cts = new CancellationTokenSource();
ConsoleCancelEventHandler onCancel = (_, e) =>
{
@@ -115,7 +115,7 @@ public static class AuditExportHelpers
/// and <see cref="SocketsHttpHandler.ConnectTimeout"/> bounds only the TCP/TLS
/// connect phase, not a connected-but-silent server. Reuses
/// <see cref="ManagementHttpClient.DefaultConnectTimeout"/> as the standard CLI
/// per-call default (arch-review 2026-08-19 P6-8).
/// per-call default.
/// </summary>
public static readonly TimeSpan DefaultHeadersTimeout = ManagementHttpClient.DefaultConnectTimeout;
@@ -126,7 +126,7 @@ public static class AuditExportHelpers
/// hung/crashed server holding the connection open) is bounded rather than
/// hanging the CLI forever — the residual gap the timeout-honesty fix
/// (<c>ManagementHttpClient.DefaultConnectTimeout</c> doc comment) left in this
/// one path (arch-review 2026-08-19 P6-8).
/// one path.
/// </summary>
public static readonly TimeSpan DefaultInactivityTimeout = ManagementHttpClient.DefaultConnectTimeout;
@@ -204,7 +204,7 @@ public static class AuditExportHelpers
// Honour the documented "authorization failure → exit 2"
// contract on the REST audit surface as well. HTTP 403 is the
// primary signal; the server may also surface UNAUTHORIZED /
// FORBIDDEN via the JSON error envelope on a non-403 status.
// FORBIDDEN via the JSON error envelope on a status other than 403.
var errorCode = TryExtractErrorCode(message);
var isAuthFailure = (int)response.StatusCode == 403
|| string.Equals(errorCode, "FORBIDDEN", StringComparison.OrdinalIgnoreCase)
@@ -404,7 +404,7 @@ public static class BundleCommands
// RunBundleCommandAsync that re-implemented URL/credential resolution and
// skipped the IsAuthorizationFailure(...) check that ExecuteCommandAsync
// enforces — a server that signalled FORBIDDEN/UNAUTHORIZED via the error
// code on a non-403 status would exit 1 instead of the documented exit 2.
// code on a status other than 403 would exit 1 instead of the documented exit 2.
// The bundle path now delegates to CommandHelpers.ExecuteCommandAsync with
// the longer BundleCommandTimeout and a per-command success handler, so the
// exit-code contract is unified across every command group.
@@ -93,6 +93,7 @@ internal static class CommandHelpers
// Every alias of a deploy contract shares the same policy. The server bounds
// the operation; a shorter client deadline can report failure while it applies.
var requestTimeout = timeout ?? (command is MgmtDeployInstanceCommand or MgmtDeployArtifactsCommand or MgmtDeploySiteCommand
or MgmtDeployAreaCommand
? Timeout.InfiniteTimeSpan
: TimeSpan.FromSeconds(30));
var response = await client.SendCommandAsync(commandName, command, requestTimeout);
@@ -262,7 +263,7 @@ internal static class CommandHelpers
var objectItems = items.Where(i => i.ValueKind == JsonValueKind.Object).ToList();
// A "note" property that appears ONLY on single-property rows (a trailing
// run-level note, or — Task 14 review, 2026-09-09 — a step's folded log/exception
// run-level note, or a step's folded log/exception
// line) is not a real column: every other row's cell there would render blank,
// and column padding would swallow the raw line's own leading indentation. Such
// rows print as a raw unpadded line instead, at their position in the sequence,
@@ -1,7 +1,6 @@
using System.CommandLine;
using System.CommandLine.Parsing;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using Microsoft.AspNetCore.SignalR.Client;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
@@ -90,12 +89,49 @@ public static class DebugCommands
private static async Task<int> StreamDebugAsync(string baseUrl, string username, string password, int instanceId, string format)
{
var hubUrl = baseUrl.TrimEnd('/') + "/hubs/debug-stream";
var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}"));
// Exchange the credentials for a bearer token once (review 2026-10-21 05/U1, 05/P1) so
// the hub handshake — and every automatic reconnect — costs no LDAP bind. The hub
// authenticates through ManagementAuthenticator, which accepts the token in the
// Authorization header; the SignalR .NET client sends the AccessTokenProvider value
// there on negotiate and on the WebSocket upgrade, and re-invokes the provider on each
// reconnect, so a token that expired mid-stream is re-exchanged. A server without
// bearer support (or SCADABRIDGE_AUTH_MODE=basic) keeps the Basic header.
using var authClient = new ManagementHttpClient(baseUrl, username, password);
ManagementHttpClient.BearerResult bearer;
try
{
bearer = await authClient.TryGetBearerTokenAsync(CancellationToken.None);
}
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
{
OutputFormatter.WriteError($"Connection failed: {ex.Message}", "CONNECTION_FAILED");
return 1;
}
if (bearer.Failure is not null)
{
OutputFormatter.WriteError(bearer.Failure.Error ?? "Authentication failed.", bearer.Failure.ErrorCode ?? "AUTH_FAILED");
return 1;
}
var connection = new HubConnectionBuilder()
.WithUrl(hubUrl, options =>
{
options.Headers.Add("Authorization", $"Basic {credentials}");
if (bearer.Token is not null)
{
// Re-invoked on every (re)connect. A null token means the exchange fell back
// (transient error, or the server dropped bearer support mid-stream): retry
// once, and if it is still null connect without a token — the hub refuses the
// handshake and WithAutomaticReconnect tries again on its backoff schedule.
options.AccessTokenProvider = async () =>
(await authClient.TryGetBearerTokenAsync(CancellationToken.None)).Token
?? (await authClient.TryGetBearerTokenAsync(CancellationToken.None)).Token;
}
else
{
options.Headers.Add("Authorization", $"Basic {authClient.BasicCredentials}");
}
})
.WithAutomaticReconnect()
.Build();
@@ -21,6 +21,7 @@ public static class DeployCommands
command.Add(BuildInstance(urlOption, formatOption, usernameOption, passwordOption));
command.Add(BuildSite(urlOption, formatOption, usernameOption, passwordOption));
command.Add(BuildArea(urlOption, formatOption, usernameOption, passwordOption));
command.Add(BuildArtifacts(urlOption, formatOption, usernameOption, passwordOption));
command.Add(BuildStatus(urlOption, formatOption, usernameOption, passwordOption));
@@ -84,12 +85,59 @@ public static class DeployCommands
return cmd;
}
/// <summary>
/// Builds <c>deploy area</c> — bulk-deploy every instance in one area subtree
/// (the area plus all of its descendant areas), Gitea #40.
///
/// <para>
/// <c>deploy site</c> narrowed to an area: same skip rules (Disabled always,
/// up-to-date unless <c>--force</c>), same not-all-or-nothing summary, same
/// server-owned deadline and the same exit-code contract.
/// </para>
/// </summary>
private static Command BuildArea(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
{
var idOption = new Option<int>("--id") { Description = "Area ID (its whole subtree is deployed)", Required = true };
var forceOption = new Option<bool>("--force")
{
Description =
"Redeploy instances whose config is unchanged (default: skip them — a redeploy " +
"resets static overrides and native alarm state at the site)",
};
var cmd = new Command("area") { Description = "Deploy every deployable instance in an area and its descendant areas" };
cmd.Add(idOption);
cmd.Add(forceOption);
cmd.SetAction(async (ParseResult result) =>
{
var areaId = result.GetValue(idOption);
var force = result.GetValue(forceOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new MgmtDeployAreaCommand(areaId, force),
timeout: BulkDeployTimeout,
onSuccess: json => HandleSiteDeploymentResponse(
json, CommandHelpers.ResolveFormat(result, formatOption, CliConfig.Load()), "area deployment"));
});
return cmd;
}
/// <summary>
/// The server owns the bounded overall bulk deadline; do not abandon it with a shorter client timer.
/// </summary>
internal static readonly TimeSpan BulkDeployTimeout = Timeout.InfiniteTimeSpan;
internal static int HandleSiteDeploymentResponse(string json, string format)
/// <summary>
/// Renders a deployment summary and turns its <c>failureCount</c> into the exit code:
/// 0 when every site/instance succeeded, 1 when any failed. Shared by <c>deploy site</c>,
/// <c>deploy area</c> and both artifact-deploy commands (<c>deploy artifacts</c>, <c>site deploy-artifacts</c>),
/// whose summaries all carry <c>failureCount</c> — a per-site failure is a successful HTTP
/// call, so without this the process exited 0 and scripts could not detect it.
/// </summary>
/// <param name="json">The JSON response body from the management API.</param>
/// <param name="format">The output format (e.g. json or table).</param>
/// <param name="summaryName">Display name for the summary, used in the invalid-response error.</param>
/// <returns>0 when nothing failed; 1 when any failure was reported or the summary was invalid.</returns>
internal static int HandleSiteDeploymentResponse(string json, string format, string summaryName = "site deployment")
{
CommandHelpers.HandleResponse(new ManagementResponse(200, json, null, null), format);
try
@@ -100,7 +148,7 @@ public static class DeployCommands
return count == 0 ? 0 : 1;
}
catch (JsonException) { }
OutputFormatter.WriteError("The server returned an invalid site deployment summary.", "INVALID_RESPONSE");
OutputFormatter.WriteError($"The server returned an invalid {summaryName} summary.", "INVALID_RESPONSE");
return 1;
}
@@ -113,7 +161,9 @@ public static class DeployCommands
{
var siteId = result.GetValue(siteIdOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId));
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId),
onSuccess: json => HandleSiteDeploymentResponse(
json, CommandHelpers.ResolveFormat(result, formatOption, CliConfig.Load()), "artifact deployment"));
});
return cmd;
}
@@ -215,6 +215,8 @@ public static class FlowDebugCommands
/// means "no parameters". Throws <see cref="JsonException"/> on malformed input or a
/// non-object root (the inbound API body is always a flat JSON object).
/// </summary>
/// <param name="json">The raw <c>--params</c> JSON text, or null/blank for none.</param>
/// <returns>The parsed root object, or null when no parameters were supplied.</returns>
internal static JsonElement? ParseParameters(string? json)
{
if (string.IsNullOrWhiteSpace(json))
@@ -45,7 +45,7 @@ internal static class FlowDebugTableProjection
/// <param name="showLogs">
/// When true, prints each step's <c>ScriptLog</c> lines and (on a failed step) the first
/// three lines of its exception detail as additional indented <c>note</c> rows immediately
/// following that step's row (step-logs design 2026-09-09, CLI Task 14). The <c>logs</c>
/// following that step's row. The <c>logs</c>
/// column (a count of attached log lines) is always present, independent of this flag.
/// </param>
/// <returns>Compact JSON suitable for table rendering.</returns>
@@ -103,7 +103,7 @@ internal static class FlowDebugTableProjection
if (truncated)
notes.Add("event list truncated — the audit rows remain complete (audit tree --execution-id <root>)");
// Final review M8: the page shows this; the CLI silently dropped it.
// The page shows this; the CLI silently dropped it.
if (logsTruncated)
notes.Add("inline log lines capped — replay pulls the rest");
@@ -222,8 +222,11 @@ public static class SiteCommands
cmd.SetAction(async (ParseResult result) =>
{
var siteId = result.GetValue(siteIdOption);
// Exit 1 when any site failed (see DeployCommands.HandleSiteDeploymentResponse).
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId));
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId),
onSuccess: json => DeployCommands.HandleSiteDeploymentResponse(
json, CommandHelpers.ResolveFormat(result, formatOption, CliConfig.Load()), "artifact deployment"));
});
return cmd;
}
@@ -1,12 +1,59 @@
using System.Net;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.CLI;
/// <summary>
/// HTTP client for the central management surfaces (<c>POST /management</c>, the audit REST
/// endpoints). Authenticates with a bearer token exchanged once per process at
/// <c>POST /auth/token</c>, falling back to HTTP Basic when the server does not offer bearer
/// tokens (review 2026-10-21 05/U1, 05/P1).
/// </summary>
/// <remarks>
/// <para>
/// <b>Bearer flow.</b> The first call a process makes for a given URL + credential pair posts
/// the credentials to <c>/auth/token</c> (one LDAP bind) and keeps the returned JWT in memory
/// (<see cref="CliTokenStore"/>; never persisted or logged). Every later call — from this or any
/// other client instance in the same process — sends <c>Authorization: Bearer</c>, which the
/// server validates without LDAP. A token nearing expiry is re-exchanged. A bearer call
/// answered <c>401</c> re-exchanges once (unless the token was minted for that very call) and,
/// if still refused, retries with Basic.
/// </para>
/// <para>
/// <b>Basic fallback.</b> A server whose <c>/auth/token</c> is absent (404/405), disabled
/// (login-disabled deployment, 404) or does not advertise <c>management_bearer: true</c> (an
/// older server, whose tokens no management surface accepts) is remembered as Basic-only for
/// the rest of the process. A transient exchange failure (5xx) uses Basic for that call only.
/// Bad credentials (<c>401</c>) and a throttle lockout (<c>429</c>) from the exchange are
/// returned as-is — retrying them with Basic would only add a second failed bind against the
/// login throttle.
/// </para>
/// <para>
/// <c>SCADABRIDGE_AUTH_MODE=basic</c> disables the exchange (Basic on every call).
/// </para>
/// </remarks>
public class ManagementHttpClient : IDisposable
{
/// <summary>Environment variable selecting the auth mode: <c>token</c> (default) or <c>basic</c>.</summary>
public const string AuthModeEnvVar = "SCADABRIDGE_AUTH_MODE";
/// <summary>
/// Environment variable that, when <c>1</c>/<c>true</c>, makes the client note on stderr
/// when it abandons a refused bearer token for HTTP Basic (e.g. central nodes configured
/// with different signing keys). The CLI has no other verbose switch.
/// </summary>
public const string VerboseEnvVar = "SCADABRIDGE_VERBOSE";
/// <summary>Re-exchange a token this long before its stated expiry.</summary>
internal static readonly TimeSpan TokenRefreshMargin = TimeSpan.FromSeconds(60);
private readonly HttpClient _httpClient;
private readonly AuthenticationHeaderValue _basic;
private readonly string _username;
private readonly string _password;
private readonly CliTokenStore.Session? _session;
/// <summary>
/// Default bound on the CONNECT phase only (30 s) — how long a TCP/TLS
@@ -77,14 +124,33 @@ public class ManagementHttpClient : IDisposable
/// <param name="password">The password for HTTP Basic authentication.</param>
public ManagementHttpClient(string baseUrl, string username, string password)
: this(
new HttpClient(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout() })
{
Timeout = Timeout.InfiniteTimeSpan
},
baseUrl, username, password)
CreateDefaultHttpClient(),
baseUrl, username, password,
IsBasicAuthModeForced() ? null : CliTokenStore.Shared)
{
}
/// <summary>
/// Builds the production <see cref="HttpClient"/>: infinite overall timeout (each call
/// carries its own deadline), connect bounded by <see cref="ResolveConnectTimeout"/>, and
/// redirects NOT followed — a 307/308 from a proxy would otherwise re-POST the
/// <c>/auth/token</c> form, i.e. the credentials, to wherever the redirect points.
/// </summary>
/// <returns>The configured client.</returns>
internal static HttpClient CreateDefaultHttpClient() =>
new(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout(), AllowAutoRedirect = false })
{
Timeout = Timeout.InfiniteTimeSpan
};
/// <summary>
/// True when <see cref="AuthModeEnvVar"/> is <c>basic</c>, which disables the token
/// exchange for the process.
/// </summary>
/// <returns>Whether HTTP Basic is forced on every call.</returns>
internal static bool IsBasicAuthModeForced() =>
string.Equals(Environment.GetEnvironmentVariable(AuthModeEnvVar)?.Trim(), "basic", StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Test-only constructor that accepts a pre-built <see cref="HttpClient"/> (typically
/// over a stub <see cref="HttpMessageHandler"/>) so the request/response handling can
@@ -94,16 +160,260 @@ public class ManagementHttpClient : IDisposable
/// <param name="baseUrl">The base URL for the management API.</param>
/// <param name="username">The username for HTTP Basic authentication.</param>
/// <param name="password">The password for HTTP Basic authentication.</param>
internal ManagementHttpClient(HttpClient httpClient, string baseUrl, string username, string password)
/// <param name="tokenStore">
/// Bearer-token store to exchange credentials through; <see langword="null"/> (the default)
/// sends HTTP Basic on every call, which keeps stub-handler tests to a single request.
/// </param>
internal ManagementHttpClient(HttpClient httpClient, string baseUrl, string username, string password, CliTokenStore? tokenStore = null)
{
_httpClient = httpClient;
// Test seam (WP2.6e): exposes the constructed HttpClient's effective Timeout
// Test seam: exposes the constructed HttpClient's effective Timeout
// without requiring reflection.
EffectiveTimeout = httpClient.Timeout;
_httpClient.BaseAddress = new Uri(baseUrl.TrimEnd('/') + "/");
var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}"));
_httpClient.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("Basic", credentials);
_username = username;
_password = password;
_basic = new AuthenticationHeaderValue(
"Basic", Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}")));
_session = tokenStore?.For(baseUrl, username, password);
}
/// <summary>The HTTP Basic credential value (base64 <c>user:password</c>), for the debug-stream hub fallback.</summary>
internal string BasicCredentials => _basic.Parameter!;
/// <summary>Outcome of <see cref="TryGetBearerTokenAsync"/>.</summary>
/// <param name="Token">The bearer token, or <see langword="null"/> when the caller should use Basic.</param>
/// <param name="Failure">A credential/throttle failure from the exchange that must be reported, not retried with Basic.</param>
internal readonly record struct BearerResult(string? Token, ManagementResponse? Failure);
/// <summary>
/// Returns a usable bearer token for this client's credentials, exchanging at
/// <c>/auth/token</c> if none is cached — used by the debug-stream hub, which cannot replay a
/// rejected handshake. <see cref="BearerResult.Token"/> is null when the caller should fall
/// back to Basic; <see cref="BearerResult.Failure"/> is set for bad credentials or a lockout.
/// </summary>
/// <param name="cancellationToken">Cancels the exchange.</param>
/// <returns>The token, a fall-back-to-Basic signal, or a failure to report.</returns>
internal async Task<BearerResult> TryGetBearerTokenAsync(CancellationToken cancellationToken)
{
var auth = await AcquireAuthAsync(forceExchange: false, refusedToken: null, cancellationToken).ConfigureAwait(false);
if (auth.Failure is not null)
{
var failure = await ToManagementResponseAsync(auth.Failure, cancellationToken).ConfigureAwait(false);
auth.Failure.Dispose();
return new BearerResult(null, failure);
}
return new BearerResult(auth.Header.Scheme == "Bearer" ? auth.Header.Parameter : null, null);
}
private readonly record struct AuthChoice(AuthenticationHeaderValue Header, bool FreshlyMinted, HttpResponseMessage? Failure);
/// <summary>
/// Picks the Authorization header for the next request: the cached bearer token, a freshly
/// exchanged one, or Basic. A non-null <see cref="AuthChoice.Failure"/> is the exchange's own
/// 401/429 response, to be surfaced to the caller as-is.
/// </summary>
/// <param name="forceExchange">Exchange even though a cached token looks usable (the server refused it).</param>
/// <param name="refusedToken">
/// The token the server just refused. If another caller has already replaced it by the time
/// the gate is acquired, that replacement is returned instead of exchanging again.
/// </param>
/// <param name="ct">Cancellation token.</param>
private async Task<AuthChoice> AcquireAuthAsync(bool forceExchange, string? refusedToken, CancellationToken ct)
{
var session = _session;
if (session is null || session.BasicOnly)
return new AuthChoice(_basic, false, null);
try
{
await session.Gate.WaitAsync(ct).ConfigureAwait(false);
}
catch (OperationCanceledException ex) when (ex is not TaskCanceledException)
{
// Callers map TaskCanceledException to their TIMEOUT result, as for the send itself.
throw new TaskCanceledException(ex.Message, ex, ct);
}
try
{
if (session.BasicOnly)
return new AuthChoice(_basic, false, null);
if (!forceExchange && session.HasUsableToken(DateTimeOffset.UtcNow))
return new AuthChoice(new AuthenticationHeaderValue("Bearer", session.Token), false, null);
if (forceExchange && session.HasUsableToken(DateTimeOffset.UtcNow)
&& !string.Equals(session.Token, refusedToken, StringComparison.Ordinal))
{
// A concurrent caller already re-exchanged: use its token (as freshly minted, so a
// second refusal falls back to Basic rather than exchanging yet again).
return new AuthChoice(new AuthenticationHeaderValue("Bearer", session.Token), true, null);
}
session.Token = null;
using var form = new FormUrlEncodedContent(
[
new KeyValuePair<string, string>("username", _username),
new KeyValuePair<string, string>("password", _password),
]);
var response = await _httpClient.PostAsync("auth/token", form, ct).ConfigureAwait(false);
if (response.StatusCode is HttpStatusCode.Unauthorized or HttpStatusCode.TooManyRequests)
return new AuthChoice(_basic, false, response);
using (response)
{
if (response.StatusCode is HttpStatusCode.NotFound or HttpStatusCode.MethodNotAllowed)
{
session.BasicOnly = true;
return new AuthChoice(_basic, false, null);
}
if (!response.IsSuccessStatusCode)
return new AuthChoice(_basic, false, null); // transient: Basic for this call only
var body = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
if (!TryParseToken(body, out var token, out var lifetime))
{
// An older server: /auth/token exists but no management surface takes its tokens.
session.BasicOnly = true;
return new AuthChoice(_basic, false, null);
}
session.Token = token;
var margin = lifetime > TokenRefreshMargin * 2 ? TokenRefreshMargin : TimeSpan.Zero;
session.RefreshAfter = DateTimeOffset.UtcNow + lifetime - margin;
return new AuthChoice(new AuthenticationHeaderValue("Bearer", token), true, null);
}
}
finally
{
session.Gate.Release();
}
}
private static bool TryParseToken(string body, out string token, out TimeSpan lifetime)
{
token = string.Empty;
lifetime = TimeSpan.FromMinutes(15);
try
{
using var doc = JsonDocument.Parse(body);
var root = doc.RootElement;
if (root.ValueKind != JsonValueKind.Object
|| !root.TryGetProperty("management_bearer", out var bearer) || bearer.ValueKind != JsonValueKind.True
|| !root.TryGetProperty("access_token", out var accessToken) || accessToken.ValueKind != JsonValueKind.String
|| string.IsNullOrWhiteSpace(accessToken.GetString()))
{
return false;
}
token = accessToken.GetString()!;
if (root.TryGetProperty("expires_in", out var expiresIn) && expiresIn.TryGetInt32(out var seconds) && seconds > 0)
lifetime = TimeSpan.FromSeconds(seconds);
return true;
}
catch (JsonException)
{
return false;
}
}
/// <summary>
/// Sends a request built by <paramref name="buildRequest"/> with the chosen credentials,
/// applying the bearer refresh-once-then-Basic policy on a 401.
/// </summary>
private async Task<HttpResponseMessage> SendAuthenticatedAsync(
Func<HttpRequestMessage> buildRequest, HttpCompletionOption completion, CancellationToken ct)
{
var auth = await AcquireAuthAsync(forceExchange: false, refusedToken: null, ct).ConfigureAwait(false);
if (auth.Failure is not null)
return auth.Failure;
var response = await SendWithAsync(buildRequest, auth.Header, completion, ct).ConfigureAwait(false);
if (response.StatusCode != HttpStatusCode.Unauthorized || auth.Header.Scheme != "Bearer")
return response;
// The server refused our bearer token. A token cached from an earlier call may simply be
// stale (expired, signing key rotated): re-exchange once and retry.
if (!auth.FreshlyMinted)
{
response.Dispose();
auth = await AcquireAuthAsync(forceExchange: true, refusedToken: auth.Header.Parameter, ct).ConfigureAwait(false);
if (auth.Failure is not null)
return auth.Failure;
response = await SendWithAsync(buildRequest, auth.Header, completion, ct).ConfigureAwait(false);
if (response.StatusCode != HttpStatusCode.Unauthorized || auth.Header.Scheme != "Bearer")
return response;
}
// A token minted moments ago is still refused (e.g. the node that answered does not share
// the signing key): fall back to Basic and stop exchanging for this process.
response.Dispose();
response = await SendWithAsync(buildRequest, _basic, completion, ct).ConfigureAwait(false);
if (response.StatusCode != HttpStatusCode.Unauthorized && _session is not null)
{
await MarkBasicOnlyAsync(_session, ct).ConfigureAwait(false);
if (IsVerbose())
{
Console.Error.WriteLine(
"note: the server refused a freshly issued bearer token; using HTTP Basic for the rest of this run "
+ "(check that both central nodes share ScadaBridge:Security:JwtSigningKey).");
}
}
return response;
}
// Written under the session gate so it cannot interleave with an in-flight exchange.
private static async Task MarkBasicOnlyAsync(CliTokenStore.Session session, CancellationToken ct)
{
await session.Gate.WaitAsync(ct).ConfigureAwait(false);
try
{
session.BasicOnly = true;
session.Token = null;
}
finally
{
session.Gate.Release();
}
}
private static bool IsVerbose()
{
var value = Environment.GetEnvironmentVariable(VerboseEnvVar)?.Trim();
return value == "1" || string.Equals(value, "true", StringComparison.OrdinalIgnoreCase);
}
private Task<HttpResponseMessage> SendWithAsync(
Func<HttpRequestMessage> buildRequest, AuthenticationHeaderValue header, HttpCompletionOption completion, CancellationToken ct)
{
var request = buildRequest();
request.Headers.Authorization = header;
return _httpClient.SendAsync(request, completion, ct);
}
private static async Task<ManagementResponse> ToManagementResponseAsync(HttpResponseMessage response, CancellationToken ct)
{
var responseBody = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
return new ManagementResponse((int)response.StatusCode, responseBody, null, null);
string? error;
string? code = null;
try
{
using var doc = JsonDocument.Parse(responseBody);
error = doc.RootElement.TryGetProperty("error", out var e) ? e.GetString() : responseBody;
code = doc.RootElement.TryGetProperty("code", out var c) ? c.GetString() : null;
}
catch
{
error = responseBody;
}
return new ManagementResponse((int)response.StatusCode, null, error, code);
}
/// <summary>
@@ -120,12 +430,15 @@ public class ManagementHttpClient : IDisposable
var body = JsonSerializer.Serialize(new { command = commandName, payload },
new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
var content = new StringContent(body, Encoding.UTF8, "application/json");
HttpResponseMessage httpResponse;
try
{
httpResponse = await _httpClient.PostAsync("management", content, cts.Token);
httpResponse = await SendAuthenticatedAsync(
() => new HttpRequestMessage(HttpMethod.Post, "management")
{
Content = new StringContent(body, Encoding.UTF8, "application/json"),
},
HttpCompletionOption.ResponseContentRead, cts.Token);
}
catch (TaskCanceledException)
{
@@ -165,7 +478,7 @@ public class ManagementHttpClient : IDisposable
/// <c>/api/audit/query</c> endpoint) and returns the
/// response body. Unlike <see cref="SendCommandAsync"/>, this does not wrap the call
/// in the <c>POST /management</c> command envelope — the audit endpoints are plain
/// REST resources. Authentication (HTTP Basic) and the base address are shared.
/// REST resources. Authentication (bearer token or HTTP Basic) and the base address are shared.
/// </summary>
/// <param name="relativePath">Path relative to the base URL, with query string.</param>
/// <param name="timeout">The request timeout.</param>
@@ -177,7 +490,9 @@ public class ManagementHttpClient : IDisposable
HttpResponseMessage httpResponse;
try
{
httpResponse = await _httpClient.GetAsync(relativePath, cts.Token);
httpResponse = await SendAuthenticatedAsync(
() => new HttpRequestMessage(HttpMethod.Get, relativePath),
HttpCompletionOption.ResponseContentRead, cts.Token);
}
catch (TaskCanceledException)
{
@@ -216,7 +531,7 @@ public class ManagementHttpClient : IDisposable
/// maintenance endpoints) with a JSON body and returns the response. Unlike
/// <see cref="SendCommandAsync"/>, this does not wrap the call in the
/// <c>POST /management</c> command envelope — these are plain REST resources.
/// Authentication (HTTP Basic) and the base address are shared.
/// Authentication (bearer token or HTTP Basic) and the base address are shared.
/// </summary>
/// <param name="relativePath">Path relative to the base URL.</param>
/// <param name="body">The JSON body to send, or <c>null</c> for an empty body.</param>
@@ -226,12 +541,15 @@ public class ManagementHttpClient : IDisposable
{
using var cts = new CancellationTokenSource(timeout);
var content = new StringContent(body ?? "{}", Encoding.UTF8, "application/json");
HttpResponseMessage httpResponse;
try
{
httpResponse = await _httpClient.PostAsync(relativePath, content, cts.Token);
httpResponse = await SendAuthenticatedAsync(
() => new HttpRequestMessage(HttpMethod.Post, relativePath)
{
Content = new StringContent(body ?? "{}", Encoding.UTF8, "application/json"),
},
HttpCompletionOption.ResponseContentRead, cts.Token);
}
catch (TaskCanceledException)
{
@@ -276,7 +594,9 @@ public class ManagementHttpClient : IDisposable
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
/// <returns>The raw HTTP response message for streaming.</returns>
public async Task<HttpResponseMessage> SendGetStreamAsync(string relativePath, CancellationToken cancellationToken)
=> await _httpClient.GetAsync(relativePath, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
=> await SendAuthenticatedAsync(
() => new HttpRequestMessage(HttpMethod.Get, relativePath),
HttpCompletionOption.ResponseHeadersRead, cancellationToken);
/// <summary>
/// Disposes the underlying HTTP client.
@@ -36,8 +36,7 @@ public static class OutputFormatter
/// <summary>
/// One row for the <see cref="TableRow"/> overload of <c>WriteTable</c>: either padded
/// column cells like the plain <see cref="string[]"/> overload, or an already-formatted
/// line printed verbatim with no column padding/alignment at all (Task 14 review,
/// 2026-09-09) — used for a note/log line folded onto a step, which must render as a raw
/// line printed verbatim with no column padding/alignment at all — used for a note/log line folded onto a step, which must render as a raw
/// indented line under that step's row rather than as a cell in a table column (every
/// other row's cell there would otherwise render blank, and the line's own leading
/// whitespace would be swallowed by column padding).
@@ -50,14 +49,20 @@ public static class OutputFormatter
RawLine = rawLine;
}
/// <summary>The column cells of a normal row, or null for a raw line.</summary>
internal string[]? Cells { get; }
/// <summary>The verbatim line of a raw row, or null for a column row.</summary>
internal string? RawLine { get; }
/// <summary>A normal row, rendered as padded columns aligned with the header set.</summary>
/// <param name="cells">The column cell values.</param>
/// <returns>A column row.</returns>
internal static TableRow Columns(string[] cells) => new(cells, null);
/// <summary>An already-formatted line, printed exactly as given with no column alignment.</summary>
/// <param name="line">The preformatted line.</param>
/// <returns>A raw row.</returns>
internal static TableRow Raw(string line) => new(null, line);
}
+44 -6
View File
@@ -28,7 +28,7 @@ scadabridge --url http://central-host:5000 <command>
scadabridge --url http://localhost:9000 <command>
```
The management API is stateless (Basic Auth per request), so sticky sessions are unnecessary. Direct node URLs (`9001`/`9002`) are useful for diagnostics; use Traefik for active-node routing and failover.
The management API is stateless (a self-contained bearer token, or Basic Auth per request), so sticky sessions are unnecessary. Direct node URLs (`9001`/`9002`) are useful for diagnostics; use Traefik for active-node routing and failover.
## Global Options
@@ -61,6 +61,7 @@ For the Docker test environment, see `docker/README.md` for a ready-to-use confi
| `SCADABRIDGE_FORMAT` | Default output format (overrides config file) |
| `SCADABRIDGE_USERNAME` | LDAP username (fallback when `--username` is not supplied) |
| `SCADABRIDGE_PASSWORD` | LDAP password (fallback when `--password` is not supplied). Preferred over `--password` on the command line, which leaks into process listings and shell history. |
| `SCADABRIDGE_AUTH_MODE` | `token` (default) exchanges credentials for a bearer token once per process; `basic` sends HTTP Basic on every call (see *Authentication* below). |
## Output
@@ -572,7 +573,7 @@ scadabridge --url <url> instance list [--site-id <int>] [--template-id <int>] [-
|--------|----------|-------------|
| `--site-id` | no | Filter by site ID |
| `--template-id` | no | Filter by template ID |
| `--search` | no | Search term matched against instance name |
| `--search` | no | Unique-name **prefix** (`StartsWith`; was a substring match before review 2026-10-21 08/P2) |
| `--skip` | no | Offset paging: number of items to skip (default 0). Applied after site-scope filtering. |
| `--take` | no | Offset paging: max items to return (clamped 1..1000; omit for unlimited) |
@@ -965,6 +966,11 @@ scadabridge --url <url> site deploy-artifacts [--site-id <int>]
|--------|----------|-------------|
| `--site-id` | no | Target site ID; deploys system-wide artifacts to exactly that site. Omit to deploy fleet-wide to all sites. A site-scoped (non-Administrator) Deployer must supply an in-scope `--site-id`; fleet-wide deployment requires a system-wide Deployer or Administrator (arch-review C2). |
Output is the per-site result matrix (`siteId`, `siteName`, `success`, `errorMessage`,
`secretsDeployed`, `secretsRemoved`) plus `successCount` / `failureCount`. The command exits `1`
when `failureCount` is non-zero — a site that failed (unreachable, a missing or unreadable
secret, a partial secret apply) is a failed command even though the HTTP call succeeded.
#### `site area list`
List all areas for a site.
@@ -1063,7 +1069,7 @@ and the rest proceed. Each failed instance is individually retryable with
Site round-trips run with bounded concurrency
(`ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism`, default 4), each under
`SiteDeploymentTimeoutPerInstance` (default 120 s), so one wedged instance cannot stall
the batch. The client waits up to 5 minutes.
the batch. The client waits without its own deadline; the server owns the bulk-deploy deadline.
Two classes of instance are **skipped** — not flattened, not sent, not changed in any
way — and reported in `skippedInstances` with a reason instead of counting as a success
@@ -1095,6 +1101,25 @@ Output is a per-instance result matrix (`instanceId`, `uniqueName`, `deploymentI
`success`, `errorMessage`) plus `successCount` / `failureCount`, and `skippedInstances`
(`instanceId`, `uniqueName`, `reason`).
#### `deploy area`
Deploy every deployable instance in one **area subtree** — the area itself plus every
area nested beneath it at any depth (Gitea #40). It is `deploy site` narrowed to that
subtree, with identical rules: Disabled instances are always skipped, up-to-date ones
are skipped unless `--force`, the batch is not all-or-nothing, and the output is the
same summary (`siteIdentifier` is the area's site). Instances with no area are only
reachable through `deploy site`. Same deadline policy as `deploy site`, and the same
exit-code rule: `1` when any instance failed (or the summary is invalid), else `0`.
```sh
scadabridge --url <url> deploy area --id <areaId> [--force]
```
| Option | Required | Description |
|--------|----------|-------------|
| `--id` | yes | Area ID. A site-scoped (non-Administrator) Deployer must name an area of an in-scope site; an unknown area is denied to them exactly like an out-of-scope one. |
| `--force` | no | Redeploy instances whose config is unchanged (default: skip them). Disabled instances are still skipped. |
#### `deploy artifacts`
Deploy compiled artifacts to one or all sites (same as `site deploy-artifacts`).
@@ -1107,6 +1132,8 @@ scadabridge --url <url> deploy artifacts [--site-id <int>]
|--------|----------|-------------|
| `--site-id` | no | Target site ID; deploys to exactly that site. Omit to deploy fleet-wide. A site-scoped (non-Administrator) Deployer must supply an in-scope `--site-id`; fleet-wide requires a system-wide Deployer or Administrator (arch-review C2). |
Same output and exit-code rule as `site deploy-artifacts`: exit `1` when any site failed.
#### `deploy status`
Query deployment records, with optional filters and pagination.
@@ -2657,7 +2684,7 @@ run's outbound API / DB / notification rows, which are always audited.
## Deployment deadlines
`deploy instance`, `instance deploy`, `deploy artifacts`, `site deploy-artifacts`,
and `deploy site` defer the overall deadline to the Management API. They keep the
`deploy site` and `deploy area` defer the overall deadline to the Management API. They keep the
bounded socket-connect timeout but do not impose the ordinary 30-second client
request deadline on a deployment that may still be applying. The API's configured
long-running timeout remains authoritative. Ordinary commands retain a 30-second
@@ -2665,13 +2692,24 @@ client deadline; explicit bundle/Flow Debugger timeouts retain their own policie
## Architecture Notes
The CLI connects to the Central cluster over **HTTP** (`ManagementHttpClient`), posting to the `/management` endpoints at the configured `managementUrl` — normally the Traefik load balancer, which routes to the active central node. It carries HTTP **Basic** credentials from `--username`/`--password`. Central's endpoint handler asks the `ManagementActor` in-process via `ManagementActorHolder`.
The CLI connects to the Central cluster over **HTTP** (`ManagementHttpClient`), posting to the `/management` endpoints at the configured `managementUrl` — normally the Traefik load balancer, which routes to the active central node. It authenticates with a bearer token (see *Authentication*). Central's endpoint handler asks the `ManagementActor` in-process via `ManagementActorHolder`.
There is no Akka dependency in the CLI at all: it does not join the cluster, does not use `ClusterClient`, and does not contact a `ClusterClientReceptionist`. (Earlier revisions of this document described a ClusterClient transport that was never built.)
An `HttpClient` is created per command invocation and disposed when the command completes.
Role enforcement is applied by the ManagementActor on the server side. The CLI sends `--username` / `--password` as HTTP Basic credentials. The server performs LDAP authentication, resolves group memberships, and maps them to roles such as Administrator, Designer, and Deployer. Operations require the appropriate role — for example, creating templates requires `Designer`, deploying requires `Deployer`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles.
Role enforcement is applied by the ManagementActor on the server side. The server performs LDAP authentication (once, at the token exchange — or per call on the Basic fallback), resolves group memberships, and maps them to roles such as Administrator, Designer, and Deployer. Operations require the appropriate role — for example, creating templates requires `Designer`, deploying requires `Deployer`. In the test environment, use the `multi-role` user (password: `password`) which has all three roles.
## Authentication
The CLI resolves `--username` / `--password` (or `SCADABRIDGE_USERNAME` / `SCADABRIDGE_PASSWORD`) and, on the first management call of the process, exchanges them at `POST /auth/token` for a short-lived bearer JWT (default 15 minutes) carrying the username, roles and permitted sites. That exchange is the invocation's only LDAP bind: every later call — `/management`, the audit REST endpoints, and `debug stream` (including its reconnects) — sends `Authorization: Bearer <token>`, which the server validates without contacting LDAP, so a short LDAP outage does not interrupt a run that already holds a token.
- The token is kept **in memory only** (never written to disk, logged or printed) and dies with the process.
- A call refused with `401` re-exchanges once, then retries with HTTP Basic.
- A server without bearer support — no `/auth/token`, a login-disabled deployment, or an older server whose token response lacks `management_bearer: true` — is used with HTTP Basic for the rest of the process (an LDAP bind per call, as before).
- Bad credentials or a login-throttle lockout at the exchange are reported directly (exit 1) and not retried with Basic.
- Set `SCADABRIDGE_AUTH_MODE=basic` to skip the exchange entirely.
- Set `SCADABRIDGE_VERBOSE=1` to get a stderr note when the server refuses a freshly issued token and the CLI falls back to Basic (usually central nodes with different `JwtSigningKey` values).
## Issues & Missing Features
@@ -12,6 +12,7 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.SignalR.Client" />
<PackageReference Include="Microsoft.AspNetCore.SignalR.Common" />
<PackageReference Include="System.CommandLine" />
</ItemGroup>
<ItemGroup>
@@ -2,5 +2,7 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.SandboxWorker;
internal static class WorkerProgram
{
/// <summary>Runs the sandbox worker protocol loop until the host closes it.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public static Task Main() => ScriptAnalysis.SandboxWorkerProtocol.RunWorkerAsync();
}
@@ -6,9 +6,11 @@ using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.Ldap;
using ZB.MOM.WW.Auth.Abstractions.Roles;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.ScadaBridge.Security.Auth;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
@@ -18,7 +20,7 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
/// </summary>
public static class AuthEndpoints
{
/// <summary>Registers the <c>/auth/login</c>, <c>/auth/logout</c>, and <c>/auth/ping</c> endpoints on the given route builder.</summary>
/// <summary>Registers the <c>/auth/login</c>, <c>/auth/token</c>, <c>/auth/logout</c>, <c>/auth/ping</c> and <c>/auth/activity</c> endpoints on the given route builder.</summary>
/// <param name="endpoints">The route builder to add the endpoints to.</param>
/// <returns>The same <paramref name="endpoints"/> instance, for call chaining.</returns>
public static IEndpointRouteBuilder MapAuthEndpoints(this IEndpointRouteBuilder endpoints)
@@ -118,70 +120,9 @@ public static class AuthEndpoints
context.Response.Redirect("/");
});
endpoints.MapPost("/auth/token", async (HttpContext context) =>
{
var form = await context.Request.ReadFormAsync();
var username = form["username"].ToString();
var password = form["password"].ToString();
if (string.IsNullOrWhiteSpace(username) || string.IsNullOrWhiteSpace(password))
{
return Results.Json(new { error = "Username and password are required." }, statusCode: 400);
}
// Login throttle: refuse a bind while this username+IP key is locked out
// (fixed-window password-spray guard, arch-review P1/UA5).
var throttle = context.RequestServices.GetRequiredService<LoginThrottle>();
var remoteIp = context.Connection.RemoteIpAddress?.ToString();
if (throttle.IsLockedOut(username, remoteIp!))
{
return Results.Json(
new { error = "Too many failed authentication attempts. Try again later.", code = "AUTH_THROTTLED" },
statusCode: 429);
}
var ldapAuth = context.RequestServices.GetRequiredService<ILdapAuthService>();
var jwtService = context.RequestServices.GetRequiredService<JwtTokenService>();
var roleMapper = context.RequestServices.GetRequiredService<IGroupRoleMapper<string>>();
var authResult = await ldapAuth.AuthenticateAsync(username, password, context.RequestAborted);
if (!authResult.Succeeded)
{
throttle.RecordFailure(username, remoteIp!);
return Results.Json(
new { error = LdapAuthFailureMessages.ToMessage(authResult.Failure) },
statusCode: 401);
}
throttle.RecordSuccess(username, remoteIp!);
var roleMapping = await roleMapper.MapAsync(authResult.Groups, context.RequestAborted);
// Guard the opaque-Scope unwrap (review I4); see the matching note on
// /auth/login. Fall back to no site-scope rather than throwing if a future
// mapper leaves Scope null or sets a different type.
var scope = roleMapping.Scope is RoleMappingResult mapped
? mapped
: new RoleMappingResult(roleMapping.Roles, [], IsSystemWideDeployment: false);
var displayName = string.IsNullOrEmpty(authResult.DisplayName) ? username : authResult.DisplayName;
var resolvedUsername = string.IsNullOrEmpty(authResult.Username) ? username : authResult.Username;
var token = jwtService.GenerateToken(
displayName,
resolvedUsername,
roleMapping.Roles,
scope.IsSystemWideDeployment ? null : scope.PermittedSiteIds);
return Results.Json(new
{
access_token = token,
token_type = "Bearer",
username = resolvedUsername,
display_name = displayName,
roles = roleMapping.Roles,
});
}).DisableAntiforgery();
// Credential → bearer-token exchange for the CLI (review 2026-10-21 05/U1, 05/P1).
// Antiforgery is disabled: this is a non-cookie, credential-in-body endpoint.
endpoints.MapPost("/auth/token", (Delegate)HandleTokenAsync).DisableAntiforgery();
// Logout is a state-changing authenticated action: it
// keeps antiforgery validation enabled so it cannot be triggered
@@ -212,6 +153,107 @@ public static class AuthEndpoints
return endpoints;
}
/// <summary>
/// Handler for <c>POST /auth/token</c>: binds the form-posted <c>username</c>/<c>password</c>
/// against LDAP (login-throttled), maps the groups to roles and site scope, and returns a
/// short-lived HS256 bearer JWT (<see cref="SecurityOptions.JwtExpiryMinutes"/>) that
/// <c>ManagementAuthenticator</c> accepts on <c>POST /management</c>, the audit REST
/// endpoints and the debug-stream hub. The token carries the username, display name, roles
/// and permitted site ids — everything those surfaces need, so a bearer request costs no
/// LDAP bind. The response's <c>management_bearer: true</c> marks a server whose management
/// surfaces accept the token; a client that does not see it (an older server, whose tokens
/// nothing accepted) falls back to HTTP Basic.
/// </summary>
/// <remarks>
/// Under the dev/test <c>DisableLogin</c> flag the exchange answers <c>404</c>: the
/// management surfaces already authenticate every request as the dev user, so the CLI falls
/// back to Basic, and no credential-free token is ever minted that another deployment
/// sharing the signing key might honour.
/// </remarks>
/// <param name="context">The current HTTP context.</param>
/// <returns>The token JSON on success; 400/401/404/429 otherwise.</returns>
public static async Task<IResult> HandleTokenAsync(HttpContext context)
{
var disableLogin = context.RequestServices.GetService<IOptions<AuthDisableLoginOptions>>()?.Value;
if (disableLogin?.DisableLogin == true)
{
return Results.Json(
new { error = "Token exchange is unavailable while login is disabled.", code = "TOKEN_EXCHANGE_DISABLED" },
statusCode: StatusCodes.Status404NotFound);
}
if (!context.Request.HasFormContentType)
{
return Results.Json(new { error = "Username and password are required." }, statusCode: 400);
}
var form = await context.Request.ReadFormAsync(context.RequestAborted);
var username = form["username"].ToString();
var password = form["password"].ToString();
if (string.IsNullOrWhiteSpace(username) || string.IsNullOrWhiteSpace(password))
{
return Results.Json(new { error = "Username and password are required." }, statusCode: 400);
}
// Login throttle: refuse a bind while this username+IP key is locked out
// (fixed-window password-spray guard, arch-review P1/UA5).
var throttle = context.RequestServices.GetRequiredService<LoginThrottle>();
var remoteIp = context.Connection.RemoteIpAddress?.ToString();
if (throttle.IsLockedOut(username, remoteIp!))
{
return Results.Json(
new { error = "Too many failed authentication attempts. Try again later.", code = "AUTH_THROTTLED" },
statusCode: 429);
}
var ldapAuth = context.RequestServices.GetRequiredService<ILdapAuthService>();
var jwtService = context.RequestServices.GetRequiredService<JwtTokenService>();
var roleMapper = context.RequestServices.GetRequiredService<IGroupRoleMapper<string>>();
var authResult = await ldapAuth.AuthenticateAsync(username, password, context.RequestAborted);
if (!authResult.Succeeded)
{
throttle.RecordFailure(username, remoteIp!);
return Results.Json(
new { error = LdapAuthFailureMessages.ToMessage(authResult.Failure), code = "AUTH_FAILED" },
statusCode: 401);
}
throttle.RecordSuccess(username, remoteIp!);
var roleMapping = await roleMapper.MapAsync(authResult.Groups, context.RequestAborted);
// Guard the opaque-Scope unwrap (review I4); see the matching note on
// /auth/login. Fall back to no site-scope rather than throwing if a future
// mapper leaves Scope null or sets a different type.
var scope = roleMapping.Scope is RoleMappingResult mapped
? mapped
: new RoleMappingResult(roleMapping.Roles, [], IsSystemWideDeployment: false);
var displayName = string.IsNullOrEmpty(authResult.DisplayName) ? username : authResult.DisplayName;
var resolvedUsername = string.IsNullOrEmpty(authResult.Username) ? username : authResult.Username;
// Site ids are stamped exactly as ManagementAuthenticator's Basic path projects them:
// none for a system-wide grant, the permitted set otherwise.
var token = jwtService.GenerateToken(
displayName,
resolvedUsername,
roleMapping.Roles,
scope.IsSystemWideDeployment ? null : scope.PermittedSiteIds);
return Results.Json(new
{
access_token = token,
token_type = "Bearer",
expires_in = (int)jwtService.TokenLifetime.TotalSeconds,
management_bearer = true,
username = resolvedUsername,
display_name = displayName,
roles = roleMapping.Roles,
});
}
/// <summary>
/// Handler for <c>GET /auth/ping</c>. Returns <c>200</c> while the caller's
/// cookie session is still valid and <c>401</c> once it has lapsed
@@ -228,6 +270,11 @@ public static class AuthEndpoints
}
/// <summary>Records real browser input only after authenticating and validating antiforgery.</summary>
/// <param name="context">The current HTTP context.</param>
/// <param name="antiforgery">Antiforgery service used to validate the request.</param>
/// <param name="tracker">Tracker that records the session activity.</param>
/// <param name="clock">Time source for the activity timestamp.</param>
/// <returns>An HTTP result indicating whether the activity was recorded.</returns>
public static async Task<IResult> HandleActivity(HttpContext context, IAntiforgery antiforgery,
SessionActivityTracker tracker, TimeProvider clock)
{
@@ -13,11 +13,16 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
public static class ClaimsPrincipalExtensions
{
/// <summary>Gets fresh mutation authority, failing closed on a mapping error.</summary>
/// <param name="provider">The authentication state provider.</param>
/// <returns>The authoritative authentication state.</returns>
public static Task<AuthenticationState> GetAuthoritativeStateAsync(this AuthenticationStateProvider provider)
=> provider is CookieAuthenticationStateProvider cookie
? cookie.GetAuthoritativeStateAsync() : provider.GetAuthenticationStateAsync();
/// <summary>Checks the current grant before invoking a directly audited application service.</summary>
/// <param name="provider">The authentication state provider.</param>
/// <param name="role">The role the current session must hold.</param>
/// <returns>The username of the authorized principal.</returns>
public static async Task<string> RequireRoleAsync(this AuthenticationStateProvider provider, string role)
{
var principal = (await provider.GetAuthoritativeStateAsync()).User;
@@ -15,27 +15,61 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
/// against current server state. Fresh scopes avoid sharing the circuit's DbContext.
/// Never reads IHttpContextAccessor after construction.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="GetAuthoritativeStateAsync"/> (mutation gates, UI-13) re-maps the stored LDAP
/// groups on every call and fails closed on a mapping fault.
/// </para>
/// <para>
/// <see cref="GetAuthenticationStateAsync"/> (display reads — the current username for a save
/// button, a nav-menu role check) is memoized per circuit for
/// <see cref="NonAuthoritativeRefreshInterval"/> (review 2026-10-21 07/P-3): within that window
/// it returns the state of the last revalidation, authoritative or not, instead of running the
/// role mapping again. A revoked grant can therefore linger on a display surface for at most
/// that window, while every mutation still revalidates. The underlying mapping itself is
/// served from <see cref="RoleMappingCache"/>, so even a revalidation is normally DB-free.
/// </para>
/// </remarks>
public class CookieAuthenticationStateProvider : ServerAuthenticationStateProvider
{
/// <summary>How long a non-authoritative read may reuse the circuit's last revalidated state.</summary>
public static readonly TimeSpan NonAuthoritativeRefreshInterval = TimeSpan.FromSeconds(10);
private AuthenticationState _state;
private readonly IServiceScopeFactory? _scopes;
private readonly bool _autoLogin;
private readonly TimeProvider _clock;
private readonly SemaphoreSlim _gate = new(1, 1);
// UTC ticks of the last completed revalidation; 0 = never. Read outside the gate.
private long _lastRefreshTicks;
/// <summary>Captures identity and the factory used for subsequent server-side revalidation.</summary>
public CookieAuthenticationStateProvider(IHttpContextAccessor httpContextAccessor, IServiceScopeFactory? scopes = null, IOptions<AuthDisableLoginOptions>? autoLogin = null)
/// <param name="httpContextAccessor">Supplies the user captured at construction.</param>
/// <param name="scopes">Factory for fresh scopes used during revalidation.</param>
/// <param name="autoLogin">Options indicating whether login is disabled (auto-login).</param>
/// <param name="clock">Clock for the non-authoritative memo window; defaults to the system clock.</param>
public CookieAuthenticationStateProvider(IHttpContextAccessor httpContextAccessor, IServiceScopeFactory? scopes = null, IOptions<AuthDisableLoginOptions>? autoLogin = null, TimeProvider? clock = null)
{
_state = new AuthenticationState(httpContextAccessor.HttpContext?.User ?? new ClaimsPrincipal(new ClaimsIdentity()));
_scopes = scopes;
// Trusted startup configuration, already checked by DisableLoginGuard. The
// auto-login handler deliberately uses the Cookies scheme with no LDAP groups.
_autoLogin = autoLogin?.Value.DisableLogin == true;
_clock = clock ?? TimeProvider.System;
}
/// <inheritdoc />
public override Task<AuthenticationState> GetAuthenticationStateAsync() => RefreshAsync(failClosed: false);
public override Task<AuthenticationState> GetAuthenticationStateAsync()
{
var last = Interlocked.Read(ref _lastRefreshTicks);
if (last != 0 && _clock.GetUtcNow().UtcTicks - last < NonAuthoritativeRefreshInterval.Ticks)
return Task.FromResult(Volatile.Read(ref _state));
return RefreshAsync(failClosed: false);
}
/// <summary>Requires a current mapping before a mutation; mapping faults deny rather than reuse old grants.</summary>
/// <returns>The authoritative authentication state.</returns>
public Task<AuthenticationState> GetAuthoritativeStateAsync() => RefreshAsync(failClosed: true);
private async Task<AuthenticationState> RefreshAsync(bool failClosed)
@@ -57,7 +91,8 @@ public class CookieAuthenticationStateProvider : ServerAuthenticationStateProvid
_ => _state.User,
};
changed = Authority(_state.User) != Authority(user);
_state = state = new AuthenticationState(user);
Volatile.Write(ref _state, state = new AuthenticationState(user));
Interlocked.Exchange(ref _lastRefreshTicks, _clock.GetUtcNow().UtcTicks);
}
finally { _gate.Release(); }
if (changed) NotifyAuthenticationStateChanged(Task.FromResult(state));
@@ -23,7 +23,7 @@ public static class BrowserTime
/// <param name="browserUtcOffsetMinutes">
/// The value of JavaScript <c>new Date().getTimezoneOffset()</c>: the number
/// of minutes that, <b>added</b> to local time, yields UTC. It is positive
/// for time zones behind UTC (e.g. +300 for UTC-5) and negative for zones
/// for time zones behind UTC (e.g. +300 for five hours behind UTC) and negative for zones
/// ahead (e.g. -120 for UTC+2).
/// </param>
/// <returns>The equivalent instant in UTC, or <c>null</c> when the input is null.</returns>
@@ -140,6 +140,23 @@
// the same circuit) must move this block, not leave it showing what the
// previous principal was allowed to see.
AuthStateProvider.AuthenticationStateChanged += OnAuthenticationStateChanged;
PendingUpdates.Invalidated += OnPendingUpdatesInvalidated;
}
// Set when the cache reports a finished deployment. A reload that arrives while a poll is
// already running is dropped by the PollGate, so the running poll re-checks this flag and
// goes round once more — otherwise the post-deploy count could wait out a full interval.
private int _reloadRequested;
private void OnPendingUpdatesInvalidated()
{
if (_disposed || !_visible)
{
return;
}
Interlocked.Exchange(ref _reloadRequested, 1);
_ = RunLoadAsync();
}
protected override async Task OnParametersSetAsync()
@@ -267,9 +284,20 @@
/// </summary>
internal Task PollOnceAsync() => RunLoadAsync();
private Task RunLoadAsync() => _pollGate.RunAsync(
LoadAsync,
error => Logger.LogDebug(error, "Pending-updates indicator poll tick failed."));
private async Task RunLoadAsync()
{
await _pollGate.RunAsync(
LoadAsync,
error => Logger.LogDebug(error, "Pending-updates indicator poll tick failed."));
// A reload requested while that poll was running was dropped by the gate: run once more.
if (!_disposed && Interlocked.Exchange(ref _reloadRequested, 0) == 1)
{
await _pollGate.RunAsync(
LoadAsync,
error => Logger.LogDebug(error, "Pending-updates indicator reload failed."));
}
}
private async Task LoadAsync()
{
@@ -342,6 +370,7 @@
{
_disposed = true;
AuthStateProvider.AuthenticationStateChanged -= OnAuthenticationStateChanged;
PendingUpdates.Invalidated -= OnPendingUpdatesInvalidated;
lock (_timerLock)
{
@@ -4,6 +4,7 @@
@using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit
@using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories
@attribute [Authorize(Policy = AuthorizationPolicies.OperationalAudit)]
@implements IAsyncDisposable
@inject ICentralUiRepository CentralUiRepository
@inject NavigationManager Nav
@inject IJSRuntime JS
@@ -286,6 +287,19 @@
private const string BrowserTimeModulePath = "./_content/ZB.MOM.WW.ScadaBridge.CentralUI/js/browser-time.js";
private IJSObjectReference? _browserTimeModule;
/// <summary>
/// Releases the dynamically imported browser-time module so its JS-side object reference
/// is not retained for the life of the circuit. A disconnected circuit has already
/// dropped it, so that failure is expected and swallowed.
/// </summary>
public async ValueTask DisposeAsync()
{
if (_browserTimeModule is null) return;
try { await _browserTimeModule.DisposeAsync(); }
catch (JSDisconnectedException) { /* circuit already gone */ }
_browserTimeModule = null;
}
protected override async Task OnAfterRenderAsync(bool firstRender)
{
if (!firstRender) return;
@@ -107,7 +107,7 @@
{
_siteCount = (await SiteRepository.GetAllSitesAsync()).Count;
_dataConnectionCount = (await SiteRepository.GetAllDataConnectionsAsync()).Count;
_templateCount = (await TemplateEngineRepository.GetAllTemplatesAsync()).Count;
_templateCount = (await TemplateEngineRepository.GetTemplateStubsAsync()).Count;
_apiKeyCount = (await ApiKeyAdmin.ListAsync()).Count;
}
catch
@@ -79,7 +79,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
[Inject] private ISiteRepository SiteRepository { get; set; } = default!;
/// <summary>
/// Resolves <c>{"$ref":"lib:Name"}</c> nodes in a method's parameter schema so the raw-JSON
/// escape hatch validates against the same resolved shape the form renders (#37).
/// escape hatch validates against the same resolved shape the form renders.
/// </summary>
[Inject] private ISchemaLibraryQueryService SchemaLibrary { get; set; } = default!;
[Inject] private SiteScopeService SiteScope { get; set; } = default!;
@@ -154,7 +154,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
private bool _scriptParamsValid;
/// <summary>
/// Method entry (#37): the chosen method's resolved root schema (null when it declares no
/// Method entry: the chosen method's resolved root schema (null when it declares no
/// parameters or the definitions do not parse) and its top-level fields — the card renders
/// only when there is at least one. The values live in <see cref="_methodParams"/>, the same
/// coerced value tree <c>ParameterValueForm</c> emits for the script pane, so the request
@@ -166,7 +166,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
private bool _methodParamsValid;
/// <summary>
/// Raw-JSON escape hatch for the whole method payload (#37): paste the exact body an
/// Raw-JSON escape hatch for the whole method payload: paste the exact body an
/// external caller would send. The text is parsed and schema-validated on every edit;
/// only a valid document reaches <see cref="_methodParams"/>, which stays the single source
/// of truth in both modes.
@@ -214,6 +214,8 @@ public partial class FlowDebugger : ComponentBase, IDisposable
/// on a step are NOT here — they are reported separately as
/// <see cref="FlowGraph.UnattachedLogLines"/>.
/// </summary>
/// <param name="graph">The flow graph to count log lines in, or null.</param>
/// <returns>The error, warning, info and total log-line counts.</returns>
internal static (int Errors, int Warnings, int Info, int Total) CountLogs(FlowGraph? graph)
{
if (graph is null)
@@ -706,7 +708,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
// row the run did not already have, the page is no longer short of lines and the flag
// clears. Until then it stays: the site's rows reach central's audit store by telemetry
// 5-30 s later, so a refresh that adds nothing proves nothing.
// Final review M7: read the OUTCOME's own flag — FlowDebugService copies the raw result's
// Read the OUTCOME's own flag — FlowDebugService copies the raw result's
// onto it precisely so a caller need not unwrap Raw, and reading Raw here left the record
// member with no consumer.
var logsTruncated = (_raw.LogsTruncated || outcome.LogsTruncated)
@@ -721,6 +723,9 @@ public partial class FlowDebugger : ComponentBase, IDisposable
=> events?.Count(FlowStepProjector.IsLogLine) ?? 0;
/// <summary>Preserves fresh Replay order, then appends live-only rows; duplicate event IDs use the fresh copy.</summary>
/// <param name="known">Events already held by the page, or null.</param>
/// <param name="fresh">Newly read Replay events, or null.</param>
/// <returns>The merged event list.</returns>
internal static IReadOnlyList<AuditEvent> MergeEvents(IReadOnlyList<AuditEvent>? known, IReadOnlyList<AuditEvent>? fresh)
{
var byId = new Dictionary<Guid, AuditEvent>();
@@ -852,7 +857,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
return (await SiteScope.PermittedSiteIdsAsync()).Contains(site.Id);
}
/// <inheritdoc/>
/// <summary>Cancels any pending outcome refresh when the component is disposed.</summary>
public void Dispose() => CancelPendingRefresh();
private async Task LoadReplayAsync()
@@ -972,7 +977,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
}
/// <summary>
/// Form ⇄ Raw JSON (#37). Entering raw mode seeds the editor from the current value tree so
/// Form ⇄ Raw JSON. Entering raw mode seeds the editor from the current value tree so
/// the operator edits what the form would have sent; leaving it re-mounts the form over the
/// last VALID tree (the form is keyed on the method, and it is not rendered while raw mode is
/// on, so it mounts fresh and re-reports validity from the tree).
@@ -1054,6 +1059,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
/// <c>ParameterValueForm</c> emits and what the raw editor parsed), or null when the method
/// declares no parameters. Run is gated on validity, so nothing here re-checks the tree.
/// </summary>
/// <returns>The serialized parameters, or null when the method declares none.</returns>
internal JsonElement? BuildParameters()
=> _inputFields.Count == 0 ? null : JsonSerializer.SerializeToElement(_methodParams);
@@ -1062,6 +1068,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
/// nothing. <c>ParameterValueForm</c> already emits the coerced value tree (scalars as
/// bool/long/double/string, objects as dictionaries, lists as lists), so it serializes as-is.
/// </summary>
/// <returns>The serialized parameters, or null when the form produced nothing.</returns>
internal JsonElement? BuildScriptParameters()
=> _scriptParams.Count == 0 ? null : JsonSerializer.SerializeToElement(_scriptParams);
@@ -1092,7 +1099,7 @@ public partial class FlowDebugger : ComponentBase, IDisposable
/// <summary>
/// Restores a persisted value tree only when it still validates against the method's current
/// schema — a stale tree (the schema changed, or the pre-#37 flat string-per-field entry) is
/// schema — a stale tree (the schema changed, or the older flat string-per-field entry) is
/// dropped rather than painted into the form as errors.
/// </summary>
private async Task RestoreInputsAsync(int methodId)
@@ -123,6 +123,10 @@ public partial class InstanceConfigure
/// counts: it pins a reportable template alarm off for this instance, which is a
/// real override and not the same as inheriting.
/// </summary>
/// <param name="overrideJson">The override JSON entered in the modal, if any.</param>
/// <param name="priority">The priority override, if one was chosen.</param>
/// <param name="mesReportable">The MES-reportable override, if one was chosen.</param>
/// <returns><c>true</c> when the modal carries any override worth persisting; otherwise <c>false</c>.</returns>
internal static bool HasOverrideDelta(string? overrideJson, int? priority, bool? mesReportable)
=> !string.IsNullOrWhiteSpace(overrideJson) || priority.HasValue || mesReportable.HasValue;

Some files were not shown because too many files have changed in this diff Show More