Commit Graph

427 Commits

Author SHA1 Message Date
Joseph Doherty 6f6534ac7e fix(site-runtime): artifact-apply watchdog + queued-apply failure on stop, boot-skip site event, enable revert, batch Ask slack (review 2026-10-21 lane A L1-L4)
L1: an artifact apply holding the slot past 10 min (ctor-overridable) logs an
Error site event naming it without releasing the slot; PostStop answers
queued applies Failed. L2: a boot-batch instance whose actor cannot be created
emits an Error site event and leaves the deployed set. L3: a failed enable
reverts the committed is_enabled (generation-bumped both ends). L4: the routed
SetAttributes batch Ask is the 30 s per-write bound plus 10 s mailbox slack.
2026-10-22 09:10:26 -04:00
Joseph Doherty 099b2221d6 fix(dcl): ASCII-only connection actor names, guarded ActorOf, and gate rejection of colliding connection names (review 2026-10-21 lane A M2)
ConnectionActorNames.Sanitize (shared by the DCL manager and the site deploy
gate) keeps ASCII letters/digits and -_.*+:@&=,!~';() only. The manager fails a
connection whose sanitised name is invalid or already used (Error + site
event) instead of restarting; DeployCompileValidator rejects a config whose
connection names collide after sanitising.
2026-10-22 09:10:26 -04:00
Joseph Doherty 3c11510c52 fix(site-runtime): seed republished overrides into Script Actors without firing scripts; alarms still evaluate (review 2026-10-21 01/SR-U1, decision M1)
HandleOverridesLoaded now sends children a site-local AttributeValueSeed
(stream still gets the AttributeValueChanged). ScriptActor updates its
snapshot, Conditional truth value and Expression last result (seed-only eval,
never fires); AlarmActor evaluates it as a normal change.
2026-10-22 09:10:26 -04:00
Joseph Doherty 999653a6b9 perf(site-runtime): warm the next startup batch during the stagger delay and compile a batch's distinct units in parallel (review 2026-10-21 01/SR-P3)
DeployCompileValidator.WarmCompileCache dedupes script bodies/trigger
expressions across a batch and compiles them with Parallel.ForEach (half the
cores). CreateInstanceActorBatch starts the next batch's warm as it arms the
stagger timer; creation order and delay are unchanged. The warm's parsed
configs are reused at creation.
2026-10-22 09:10:26 -04:00
Joseph Doherty dde22a3dd1 perf(site-runtime): deserialize a deploy's flattened config once and hand the parsed object down (review 2026-10-21 01/SR-P5)
The deploy compile warm parses once off-thread; the parsed FlattenedConfiguration
rides DeployCompileWarmed to the gate (new Validate(FlattenedConfiguration)
overload), the redeploy buffer, CreateInstanceActor's DCL ensure and a new
InstanceActor constructor overload. Stored-row paths parse once inside
CreateInstanceActor; an unparseable stored row still fails init via S6.
2026-10-22 09:10:26 -04:00
Joseph Doherty 960c35060c perf(site-runtime): instance reconcile sweep reads metadata only and fetches config_json just for the instances it repairs (review 2026-10-21 01/SR-P1)
Adds SiteStorageService.GetAllDeployedConfigMetadataAsync. The sweep decides
from the projection, then fetches single rows for its starts/restarts off-
thread and re-validates them (generation guard, revision match, pipeline/
running guards), releasing the per-revision claim of any repair that does
not act. Boot load unchanged.
2026-10-22 09:10:26 -04:00
Joseph Doherty c782744b07 perf(site-runtime): batch Route.To().GetAttributes/SetAttributes into one Instance Actor message (review 2026-10-21 01/SR-P4)
One Ask per attribute became one site-local InstanceGet/SetAttributesRequest
answered in a single pass, keeping the per-attribute GetAttributeResponse /
SetStaticAttributeResponse outcomes. The single-write path is refactored to a
Task<SetStaticAttributeResponse> outcome shared by both paths; static writes
and rejections still reply synchronously.
2026-10-22 09:10:26 -04:00
Joseph Doherty 35ffd025c3 fix(site-runtime): serialise DeployArtifactsCommand applies per site so an overlapping apply cannot delete a live artifact (review 2026-10-21 01/SR-C2)
Each apply upserts its full set then deletes everything else, off the actor
thread; two overlapping commands interleaved and deleted artifacts central
never removed. The Deployment Manager now runs one apply at a time, queuing
the rest in arrival order; the slot is released by a self-message that lands
behind the apply's own DCL push.
2026-10-22 09:10:26 -04:00
Joseph Doherty e765700eac fix(site-runtime): republish loaded static overrides to children/stream and answer pending waits on stop (review 2026-10-21 01/SR-U1, 01/SR-M3)
SR-U1: children are created before the async override load lands, so their
trigger snapshots kept the deploy default forever after a restart/failover/
enable/sweep start. HandleOverridesLoaded now republishes every changed value
through PublishAndNotifyChildren(evaluateWaiters:false), and skips attributes
a static write touched before the load (newer than the stored row).
SR-M3: PostStop cancels each WaitForAttribute timeout and replies
'instance stopped' instead of leaving scripts to time out and timeouts to
dead-letter.
2026-10-22 09:10:26 -04:00
Joseph Doherty f5090fcd9e fix(site-runtime): reject invalid runtime actor names at the site gate and never let ActorOf escalate to the Deployment Manager (review 2026-10-21 01/SR-C1)
An instance unique name Akka refuses as a path element (space, '#', '/',
non-ASCII) made Context.ActorOf throw on the singleton's own thread; the
default Restart stopped every Instance Actor on the site. DeployCompileValidator
now checks the instance + script/alarm/native-source actor names with
ActorPath.IsValidPathElement (the Commons ActorNameRules mirror is stricter
than Akka for " ( ), pinned by a parity sweep), CreateInstanceActor catches
InvalidActorNameException and every caller reports it, and the Instance Actor
skips a refused child name like a compile failure.
2026-10-22 09:10:26 -04:00
Joseph Doherty c55128d83d docs(centralui): 2026-10-21 review hardening (review 2026-10-21 07, 04/C4) 2026-10-22 09:00:49 -04:00
Joseph Doherty 1e3765ba6e fix(host): NodeOptions-bound ports, central ESG cache TTL, hosted warm-up wiring; document AllowSingleNodeCluster as validation-only (review 2026-10-21 08/O5, 08/O4, 05/U3) 2026-10-22 09:00:49 -04:00
Joseph Doherty fa73ddb453 fix(inbound): background active-node-only method warm-up service (review 2026-10-21 05/U3) 2026-10-22 09:00:49 -04:00
Joseph Doherty 12f815a7ea fix(esg): short-TTL definition cache for central lookups (review 2026-10-21 05/P2) 2026-10-22 09:00:49 -04:00
Joseph Doherty 37c25db2f1 fix(site-runtime): batch native-alarm deletes into the coalesced flush transaction; docs for scripts/cache/scheduler (review 2026-10-21 01/SR-P2) 2026-10-22 09:00:49 -04:00
Joseph Doherty b0a08bb63b fix(auditlog): explicit permanent rejections on the ingest ack (IngestAck.rejected_event_ids, field 2) with a dedicated counter; reconciler ignores transient faults for abandonment; persist the served high-water mark; prune only on a non-empty enumeration (review 2026-10-21 06/PERF-1, 06/IMPL-1 adversarial follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty 38167bca88 fix(auditlog): prune per-site reconciliation state and pull-channel caches for sites absent from the enumeration; bound the abandoned-id set (review 2026-10-21 06/MEM-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 9e7e19c3c1 fix(auditlog): reconciliation pull serves only unacked rows (PushAbandoned/Rejected push-exit states), central pages within a tick with set-based inserts; refused push rows leave the queue after N attempts (review 2026-10-21 06/PERF-1, 06/COR-3, 06/IMPL-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 1da5d0bac4 fix(auditlog): page the site audit purge, bound incremental_vacuum per page, fail fast instead of parking writers on a full queue (review 2026-10-21 06/MEM-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty dd60e236bd fix(communication): ReconcileSite threads the gRPC call's cancellation into central reconcile work (review 2026-10-21 03/P-1b) 2026-10-22 09:00:48 -04:00
Joseph Doherty 4456a04e81 fix(communication): failback probation is time-boxed and cleared on address change (review 2026-10-21 03/P-2 follow-up) 2026-10-22 09:00:48 -04:00
Joseph Doherty c5add11546 fix(communication): command-plane failback probes readiness with a real RPC; backoff resets only after a real call succeeds (review 2026-10-21 03/P-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 017308a397 fix(communication): DebugStreamService drops the session on an InstanceNotFound snapshot (review 2026-10-21 03/M-3) 2026-10-22 09:00:48 -04:00
Joseph Doherty 90306f0ccb fix(communication): streaming consumers re-resolve site gRPC addresses on every reopen; spec matches (review 2026-10-21 03/R-2, 03/D-2) 2026-10-22 09:00:48 -04:00
Joseph Doherty 3ed71167b3 fix(communication): site-address refresh gets an overlap guard, per-load deadline and monotonic sequence (review 2026-10-21 03/M-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty b0a97a4b67 fix(communication): malformed site gRPC address keeps the old channel, never aborts the refresh tick or restarts CentralCommunicationActor (review 2026-10-21 03/R-1) 2026-10-22 09:00:48 -04:00
Joseph Doherty 036f341a9d fix(configdb): opt-in config-audit retention purge + after-state size cap (review 2026-10-21 08/G1) 2026-10-22 08:33:09 -04:00
Joseph Doherty 394234b450 fix(dcl): advance the session generation when a reconnect attempt is issued, not when its result is applied (review 2026-10-21 02/C1 follow-up) 2026-10-22 08:33:00 -04:00
Joseph Doherty d8d44e0b87 perf(dcl): coalesce MxGateway alarm stream reopens across a subscribe burst (review 2026-10-21 02/P3) 2026-10-22 08:33:00 -04:00
Joseph Doherty e6406c494c fix(dcl): async OPC UA endpoint discovery bounded by OperationTimeoutMs/verify budget; adapter connect never runs on the actor thread (review 2026-10-21 02/P2) 2026-10-22 08:33:00 -04:00
Joseph Doherty cfe95a7e5a fix(dcl): log the initial connect to Site Event Logging like every other transition (review 2026-10-21 02/U4) 2026-10-22 08:33:00 -04:00
Joseph Doherty 3543a7f295 fix(dcl): feed ScadaBridge:MxGateway:ClientNamePrefix into the MxGateway client-name fallback (review 2026-10-21 02/U1, 08/O2) 2026-10-22 08:33:00 -04:00
Joseph Doherty f18d3adbfc fix(dcl): forget an OPC UA condition's last state once it returns to normal (review 2026-10-21 02/L3) 2026-10-22 08:33:00 -04:00
Joseph Doherty d51ea5136a fix(dcl): drop values for unsubscribed tags before quality bookkeeping; cancel the reconnect re-seed when its session ends (review 2026-10-21 02/L2, 02/L4) 2026-10-22 08:33:00 -04:00
Joseph Doherty 8c1bd2c3fc fix(dcl): advance the session generation on same-adapter reconnect and session-scope MxGateway subscription ids (review 2026-10-21 02/C1, 02/L1) 2026-10-22 08:33:00 -04:00
Joseph Doherty b811a2451b fix(dcl): remove unused poll-based WriteBatchAndWaitAsync from IDataConnection and both adapters (review 2026-10-21 02/U3) 2026-10-22 08:33:00 -04:00
Joseph Doherty 3169429c3f fix(site-event-log): queries on own read connection off the actor thread, bounded keyword window; cap trim counts/vacuums rarely (review 2026-10-21 02/P1, 02/P4) 2026-10-22 08:32:37 -04:00
Joseph Doherty f7c4adbaed docs(store-and-forward): replication text is LocalDb CDC; document lane paging + gauge reseed (review 2026-10-21 02/U4a) 2026-10-22 08:32:37 -04:00
Joseph Doherty e968566e9d fix(cli): no redirects on the management client, race-safe token refresh, verbose Basic-fallback note (review 2026-10-21 05/U1, 05/P1 L1/L2/L3/L4)
- CreateDefaultHttpClient sets AllowAutoRedirect=false so a 307/308 cannot re-POST the
  /auth/token credential form elsewhere.
- A forced re-exchange after a 401 carries the refused token; inside the gate it reuses
  a token another caller already re-exchanged instead of binding again. The BasicOnly
  switch is written under the same gate.
- SCADABRIDGE_VERBOSE=1 prints a stderr note when a fresh token is refused and the CLI
  falls back to Basic.
- debug stream's AccessTokenProvider retries the exchange once and documents the
  null-token degradation (the hub refuses, automatic reconnect retries).
- Tests: concurrent refusal reuses the peer refresh (2 exchanges, not 3); real-socket
  redirect is not followed.
2026-10-22 08:25:35 -04:00
Joseph Doherty ea172edebc fix(security): bearer token hardening follow-ups — nbf skew, TTL-0 path, cache docs, extra rejection tests (review 2026-10-21 05/U1, 05/P1 L1/L5/L6)
- JwtTokenService back-dates nbf by the 30 s validation skew so a peer central whose
  clock lags the issuer accepts a fresh token after failover.
- RoleMappingCacheSeconds = 0 routes RoleMapper to the matched-rows loader (old cost)
  instead of loading every Deployer mapping's scope rules per call.
- RoleMappingCache / SecurityOptions / Component-Security.md describe Invalidate() by
  its contract plus the TTL, not by a call site.
- Tests: HS512 with the same key, issuer/audience mismatch, lagging-peer nbf, TTL-0
  matched-only reads, refresh-vs-Invalidate generation race.
2026-10-22 08:25:35 -04:00
Joseph Doherty 1b3928a0a2 fix(security): finish the bearer-token design and cache role mappings (review 2026-10-21 05/U1, 05/P1)
- ManagementAuthenticator accepts Authorization: Bearer <jwt> alongside Basic on
  /management, the audit REST endpoints and the debug-stream hub. Tokens are
  validated locally (HS256 only, alg=none rejected, issuer/audience, lifetime with
  30 s skew); roles and site scope inside a valid token are authoritative for its
  lifetime, so a bearer call costs no LDAP bind and no role re-map.
- /auth/token returns expires_in + management_bearer:true, carries username, display
  name, roles and permitted sites, and answers 404 under DisableLogin.
- CLI exchanges credentials once per process (in-memory CliTokenStore), sends Bearer,
  re-exchanges once on 401 then falls back to Basic; older/login-disabled servers are
  Basic-only; bad credentials are not retried with Basic. Debug stream uses the token
  via AccessTokenProvider. SCADABRIDGE_AUTH_MODE=basic opts out.
- StartupValidator fails a Central boot on a JwtSigningKey under 32 UTF-8 bytes;
  fixed the false 'fail-fast at construction' comment. Removed the dead
  RefreshToken/RecordActivity/ShouldRefresh/IsIdleTimedOut members and the inert
  JwtRefreshThresholdMinutes; JwtExpiryMinutes now documented as the bearer lifetime
  (validated > 0).
- RoleMapper reads through a process-wide RoleMappingCache (RoleMappingCacheSeconds,
  default 30 s): one refresh loads the table plus every Deployer mapping's scope
  rules, Invalidate() drops it (generation-guarded against in-flight refreshes).
- Component-Security.md / Component-CLI.md / CLI README describe the real model
  (cookie claims ticket, bearer for CLI) instead of a cookie-embedded JWT.
2026-10-22 08:25:35 -04:00
Joseph Doherty 7aacc49043 fix(deployment): detach each instance's settled rows from the bulk deploy's shared unit of work after Finalize (review 2026-10-21 04/M3) 2026-10-22 08:22:42 -04:00
Joseph Doherty 32e63eaa96 fix(deployment): bulk-deploy up-to-date pre-filter uses batch hash/status reads and the operation's shared flatten session; completion log reports pre-filter cost (review 2026-10-21 04/P1) 2026-10-22 08:22:42 -04:00
Joseph Doherty df67fabe68 fix(deployment): pending-updates tick uses a slim instance projection, hash-only flatten, one untracked mismatch snapshot read and a per-tick secret lookup memo (review 2026-10-21 04/P2) 2026-10-22 08:22:42 -04:00
Joseph Doherty 7c83a4aa94 fix(reconcile): batch startup reconcile into fixed round trips, bound central work by QueryTimeout, retry failed site passes with back-off (review 2026-10-21 03/P-1) 2026-10-22 08:22:41 -04:00
Joseph Doherty d6a9a79c27 fix(deployment): purge SystemArtifactDeploymentRecords in the terminal-record retention sweep (review 2026-10-21 08/G2) 2026-10-22 08:22:41 -04:00
Joseph Doherty 0d4f7d33b9 perf(templateengine): memoise CollisionDetector's composed-module walk per check (review 2026-10-21 04/P8b) 2026-10-22 06:33:44 -04:00
Joseph Doherty 52aab37299 fix(templateengine): audit projected template/script summaries instead of the tracked graph and script bodies (review 2026-10-21 08/G1) 2026-10-22 06:33:44 -04:00
Joseph Doherty 4102922640 docs(templateengine): drop the removed staleness memo; describe the full shared-script compile gate (review 2026-10-21 04/U2, 04/U3) 2026-10-22 06:33:44 -04:00
Joseph Doherty c7774b7ec2 perf(templateengine): alarm-override, folder-delete and template-delete checks read lean template projections (review 2026-10-21 04/P6, 08/P1)
InstanceService.SetAlarmOverrideAsync and TemplateDeletionService move to the
no-tracking, body-free GetAllTemplatesForAnalysisAsync; TemplateFolderService
counts folder contents from GetTemplateStubsAsync.
2026-10-22 06:33:44 -04:00