ScadaBridge/CLAUDE.md

ScadaBridge Implementation Project

This is the full implementation project for ScadaBridge — a distributed SCADA system built on Akka.NET in a hub-and-spoke architecture (one central cluster + multiple site clusters). It contains source code, tests, deployable docker topology, and the design documentation that the code implements. The design docs are the spec; src/ is the binary.

When a change is requested, the default assumption is: update the design doc and the code and the tests and (if it ships) the docker deploy — together, in one session, with git diff review before committing.

Project Structure

Top-level directories

Layout is self-describing (ls, plus each directory's own README). Only the non-obvious parts are recorded here:

• docker/ — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with bash docker/deploy.sh.
• docker-env2/ — minimal second cluster (2 central + 1 site × 2 nodes) that runs concurrently with docker/ on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with bash docker-env2/deploy.sh.
• infra/ — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API). Traefik is not an infra/ service — it is per-rig (docker/docker-compose.yml). LDAP is no longer started here — both rigs (docker/, docker-env2/) bind the local redundant GLAuth pair (zb-ldap-primary :3893 + zb-ldap-backup :3894 on host.docker.internal; source of truth scadaproj/infra/glauth-redundant/, verified against each central node's appsettings.Central.json — arch-review 2026-08-19 P8-7), NOT the shared 10.100.0.35:3893 GLAuth, whose serviceaccount binding was rotated away by SEC-36. docs/test_infra/test_infra.md still reads as if LDAP were a local container.
• deprecated/ — retired docs/notes kept only for reference; not current.

Sister Projects

This repo is indexed by the parent workspace ~/Desktop/scadaproj, whose CLAUDE.md is the canonical umbrella index of the app family (the scadaproj repo is the umbrella for shared ZB.MOM.WW.* libs, infra/GLAuth, and the design docs the peers implement). It carries a ScadaBridge entry plus the full cross-project relationship map — consult it rather than duplicating the family list here, as it stays current.

Repos cloned as sibling directories under ~/Desktop/ — referenced for context, not part of this solution. Directly referenced by ScadaBridge:

• ~/Desktop/MxAccessGateway — MxAccess Gateway (https://gitea.dohertylan.com/dohertj2/mxaccessgw).
• ~/Desktop/OtOpcUa — OtOpcUa (https://gitea.dohertylan.com/dohertj2/lmxopcua).

Other peers in the scadaproj family (see scadaproj/CLAUDE.md for details): ~/Desktop/HistorianGateway (historiangw) — AVEVA Historian gRPC sidecar, OtOpcUa's sole historian backend; its historian SDK (AVEVA.Historian.Client) is now owned inside that repo at HistorianGateway/histsdk/ (the separate ~/Desktop/histsdk repo was folded in with history); and ~/Desktop/scadaproj (scadaproj) itself — shared ZB.MOM.WW.* libs + the dev/test GLAuth on 10.100.0.35:3893.

⛔ Wonder box remote access (2026-09-11): no psbridge, no winrm. Both skills are retired/disabled — the WinRM remote-exec channel set off the security software on the wonder boxes (vd03/vq03/vq04). Do not run remote commands (service control, publish/swap, PowerShell) on those boxes and do not stand up a replacement channel until the user clears one. Still allowed: UNC/SMB file copies and logging directly into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella ../scadaproj/CLAUDE.md for the current hold.

Document Conventions

• Requirements documents (high-level and component-level) live in docs/requirements/.
• Test infrastructure documentation lives in docs/test_infra/.
• Component documents are named Component-<Name>.md (PascalCase, hyphen-separated).
• Each component document follows a consistent structure: Purpose, Location, Responsibilities, detailed design sections, Dependencies, and Interactions.
• The README.md component table must stay in sync with actual component documents. When a component is added, removed, or renamed, update the table.
• Cross-component references in Dependencies and Interactions sections must be kept accurate across all documents. When a component's role changes, update references in all affected documents.
• Exception: DelmiaNotifier (#27) is an external client tool, not a cluster component; its spec is the project README (src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md) plus docs/plans/2026-06-26-delmia-recipe-notifier-design.md — there is intentionally no Component-DelmiaNotifier.md.

Refinement Process

• When refining requirements, use the Socratic method — ask clarifying questions before making changes.
• Probe for implications of decisions across components before updating documents.
• When a decision is made, identify all affected documents and update them together for consistency.
• After updates, verify no stale cross-references remain (e.g., references to removed or renamed components).

Editing Rules

• Edit documents and code in place. Do not create copies or backup files.
• When a change affects multiple documents or projects, update them all in the same session — design doc, entities/repos, actors/services, UI, tests, migrations, and deploy config travel together.
• Use git diff to review changes before committing.
• Commit related changes together with a descriptive message summarizing the design decision and the implementation slice.
• After non-trivial code changes, build (dotnet build ZB.MOM.WW.ScadaBridge.slnx) and run relevant tests before declaring done; for cluster-runtime changes, rebuild the image with bash docker/deploy.sh.
• Run tests with dotnet test ZB.MOM.WW.ScadaBridge.slnx. ZB.MOM.WW.ScadaBridge.CLI.Tests is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
• Propagate cross-repo changes to the umbrella index. This repo is indexed by the parent workspace ~/Desktop/scadaproj. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the ScadaBridge entry in ../scadaproj/CLAUDE.md in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. MxAccessGateway/HistorianGateway.) Keep the index row to a few lines — since 2026-09-03 the umbrella keeps only short summary rows and parks long per-project narrative in ../scadaproj/.claude/skills/sister-project-history/<project>.md (create scadabridge.md there if ScadaBridge ever needs one; OtOpcUa and HistorianGateway already have theirs) and shared-library history in ../scadaproj/<lib>/CLAUDE.md.

Component List

27 components. The catalog — what each one is and its key design commitments — lives in the scadabridge-components skill (.claude/skills/scadabridge-components/SKILL.md); the authoritative spec for each is docs/requirements/Component-<Name>.md, and README.md carries the component table.

Key Design Decisions (for context across sessions)

Architecture & Runtime

• Instance modeled as Akka actor (Instance Actor) — single source of truth for runtime state.
• Site Runtime actor hierarchy: Deployment Manager singleton → Instance Actors → Script Actors + Alarm Actors.
• Script/Alarm Actors launch script runs directly onto a dedicated blocking-I/O thread pool (ScriptExecutionScheduler) — the short-lived ScriptExecutionActor/AlarmExecutionActor children were deleted by WP3.1 (they had no Receive handler, no PostStop, and their IActorRef was never a message target; ScriptRunLauncher carries the run body verbatim). Same pool, same telemetry, same audit ExecutionId/ParentExecutionId threading, same "a stop does NOT cancel an in-flight run" semantics — minus a per-run actor cell. WP3.1 also: trigger-expression evaluation left that pool for the shared .NET thread pool behind TriggerEvalGate (arch-review finding #4 — blocked script bodies used to stall every Expression trigger on the node indefinitely); script and eval deadlines are armed at enqueue so queue wait burns the budget (a run dequeuing past its deadline is shed unrun); the pool is instance-scaled grow-only clamp(max(ScriptExecutionThreadCount, ceil(instances/8)), 1, ScriptExecutionMaxThreadCount); the stuck-script watchdog now detaches and replaces a wedged worker (capped at pool size, gauge DetachedScriptThreads); MaxConcurrentRunsPerScript (4) sheds the newest run with a counter + rate-limited site event + an explicit error reply to Ask callers; SiteScriptCompileCache evicts an oldest-⅛ batch by approximate LRU instead of clearing wholesale; and deploy/startup compiles are warmed off-thread before the (ordering-critical, now cache-hit-only) synchronous deploy gate. Design memo: docs/plans/2026-08-15-script-pool-split-design.md.
• Alarm Actors are separate peer subsystem from scripts (not inside Script Engine).
• Shared scripts execute inline as compiled code (no separate actors).
• Site-wide Akka stream for attribute value and alarm state changes with per-subscriber buffering.
• Instance Actors serialize all state mutations (Akka actor model); concurrent scripts produce interleaved side effects.
• Staggered Instance Actor startup on failover to prevent reconnection storms.
• Instance Actor tag-subscribe retry is guarded + backed off (Lever 1, 2026-08-28). One outstanding SubscribeTagsRequest per connection until TagSubscribeInFlightCeilingMs (45 s); the interval doubles ±20 % jitter from TagSubscribeRetryIntervalMs (5 s) to TagSubscribeRetryMaxIntervalMs (60 s), resetting on Success or ConnectionQualityChanged(Good). Pre-Lever-1 the actor re-sent a full N-tag subscribe every fixed 5 s with no in-flight check — under a slow MxGateway that stacked ~6 concurrent 244-tag subscribes per 30 s and wedged the session (QA 2026-08-26). Consequence to know: a lost reply is now re-sent after the ceiling (45 s), not the 5 s interval — a failed reply still retries at the backed-off interval. Levers 2 (bulk supervisory advise, cross-repo) and 3 (session command_timeout) are still design-only: docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md.
• Explicit supervision strategies: Resume for coordinator actors, Stop for short-lived execution actors.
• Level-triggered flag handshakes must observe the reset echo before raising the trigger (Gitea #36, 2026-09-10). Attributes.WriteBatchAndWaitAsync has a reset-aware overload — (values, flagKey, flagValue, responseKey, responseValue, timeout, resetKey, resetValue, resetTimeout?) → HandshakeResult (Matched/TimedOut/Phase/ResetFailed) — that writes the reset, waits event-driven for the device to echo it, and only then writes the batch + trigger and waits for the response. The six-argument bool overload is unchanged but unsafe for a repeated handshake: the waiter's current-value fast-path answers from the previous run's cached response value while the script's reset write is still in flight (QA measured 0 ms matches and stale BatchIDs). An edge-triggered wait was rejected — a subscription reports changes from the last reported value, so a device flipping reset→done inside one sampling interval never reports the reset and such a wait never resolves. Composition only: ScriptRuntimeContext.WriteBatchResetAndWait chains the existing SetAttribute → waiter → batch → waiter legs, so no actor/message/proto change, and a traced run shows all four steps. Mirrored on ScriptCompileSurface, SandboxScriptHost and FlowShapePredictor. The QA/prod MES script port is register row 119. Design: docs/plans/2026-09-10-handshake-reset-echo-design.md.

Data & Communication

• Data Connection Layer is a clean data pipe — publishes to Instance Actors only.
• DCL connection actor uses Become/Stash pattern for lifecycle state machine.
• DCL auto-reconnect at fixed interval; immediate bad quality on disconnect; transparent re-subscribe.
• DCL write failures returned synchronously to calling script.
• Tag path resolution retried periodically for devices still booting.
• Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
• Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20). Ten tables now live in ONE ZB.MOM.WW.LocalDb-managed SQLite file (the SiteLocalDbSetup.ReplicatedTables list), configured by the required LocalDb:Path (/app/data/site-localdb.db on the rig; validated with ValidateOnStart, so a site config missing it fails to boot). Consequences worth knowing:

  • site_events.id changed from autoincrement INTEGER to an application-minted GUID. Last-writer-wins keys on the primary key, so two nodes independently minting id=1,2,3… would destroy each other's events rather than merge them. The event-log read path uses a composite (timestamp, id) keyset cursor with an opaque string continuation token; EventLogEntry.Id and both ContinuationTokens are string/string? on the site↔central Akka DTOs.

  • ScadaBridge:OperationTracking:ConnectionString, ScadaBridge:SiteEventLog:DatabasePath and — as of Phase 2 — ScadaBridge:StoreAndForward:SqliteDbPath + ScadaBridge:Database:SiteDbPath are all migration-only — nothing reads them but SiteLocalDbLegacyMigrator, which copies a pre-Phase-1 file in once (deterministic mig-{NodeName}-{legacyId} event ids, INSERT OR IGNORE, runs AFTER RegisterReplicated so migrated rows replicate) and renames it .migrated. Delete the keys once a node has migrated.

  • This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths outside the mounted volume and were discarded on every container recreate.

  • Replication is default-OFF and opt-in via LocalDb:Replication:PeerAddress + a matching ApiKey on both nodes. LocalDbSyncAuthInterceptor is fail-closed: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: site-a replicated, site-b/site-c deliberately not, so both states are proven side-by-side. Status surfaces on the site health report as LocalDbReplicationConnected / LocalDbOplogBacklog (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as localdb_* Prometheus series. An UNCONFIGURED node reports null for both — the code only started matching that documented contract with register row 80: the engine is registered unconditionally and answers Connected=false/0 while inert, and publishing that made site-b/site-c look like broken replicated sites and starved central's null branches (row 33's conditional KPI emission + sustained-badge anchor exclusion) of the null they exist for. LocalDbReplicationStatusReporter now gates on a Host-supplied SiteLocalDbSetup.ReplicationIsConfigured flag, so false means exactly one thing: configured, peer session down = a real fault. Watch the trap that caused it — configured-ness is PeerAddress OR ApiKey, so "no PeerAddress" is NOT "unreplicated" (the pre-existing Host wiring test fixture is a passive-half replicated node for precisely this reason). A transient backlog spike is expected daily and is not a fault: the site_events retention purge (and the storage-cap trim) is CDC-captured like any other write — there is deliberately no purge-exemption path — so the backlog jumps by the deleted batch size and drains as the peer acks it; the purge logs an Information note saying so on a replication-enabled node (SiteEventLogReplicationCheck, supplied by the Host from SiteLocalDbSetup.ReplicationIsConfigured so SiteEventLogging never reads LocalDb config itself). Healthy = LocalDbReplicationConnected stays true and the backlog returns to ~0; see docs/deployment/topology-guide.md → Reading the replication backlog. That spike is PACED as of row 34 (2026-08-15) — ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick (default 250 000, 0 = unlimited escape hatch) is a per-tick delete ceiling shared by the retention purge and the cap trim, and a ceiling-limited tick reschedules after PurgeCatchUpInterval (default 5 min) instead of the daily PurgeInterval so retention still converges (~11 ticks ≈ 55 min for the worst case, vs 11 days without the acceleration). Measured worst case, unpaced: a 1 GB event log holds ~2.73 M rows at 393 B/row, and expiring all of them in one tick took 285 s and queued 2 731 947 oplog rows — 2.7x LocalDb's MaxOplogRows default of 1 000 000, i.e. enough to make the maintenance tick prune the oplog and flag a full ten-table snapshot resync. The subtlety that makes pacing the right fix rather than a bigger MaxOplogRows: the resident backlog is produced − drained, so a healthy peer blunts it — on a loopback pair the same 2.73 M burst peaked at only 391 k and drained in 6.0 min. The overrun is therefore conditional on peer throughput (breached outright with the peer down; breached at ~1.5 M if the peer runs at half loopback speed), and a 250 k ceiling makes the outcome independent of the peer instead. Drain speed itself is not a bottleneck (7.6–19.8 k tombstones/s measured), so a full 250 k tick clears far inside row 33's 15-minute sustained dwell — verified at 2x a tick (500 k drained in 46 s). The purge loop is now a delay loop rather than a PeriodicTimer, so its cadence anchors to the END of the previous purge. Ceiling hits log a rate-limited (15 min) Warning and always increment scadabridge.site.event_log.purge_ceiling_hits — read the counter's shape: a run that stops is a backlog converging, hits that never stop mean the ceiling is mis-sized for that site's event rate. Measurement + decision record: docs/plans/2026-08-15-site-events-purge-burst-measurement.md. Note ZbTelemetryOptions.Meters is an allowlist (SiteServiceRegistration.ObservedMeters); an unlisted meter exports nothing, silently.

  • Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under docs/plans/. Phase 2 deleted SiteReplicationActor, its ReplicationMessages, StoreAndForward's ReplicationService, and StoreAndForwardStorage.ReplaceAllAsync — do not reintroduce them:

    • CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — the standby makes no fetch at all during a deploy (SiteReconciliationActor's node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on sf_messages. ReplaceAllAsync was a destructive delete-all-then-insert-all resync and is unsafe to reintroduce: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
    • notification_lists and smtp_configurations are created but deliberately NOT registered. They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have no CDC triggers on either rig node.
    • Operational constraints (read before upgrading a site pair): stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy SfBufferSnapshot compatibility handler went with the replicator. And a node offline longer than LocalDb:Replication:TombstoneRetention (default 7 days) can resurrect deleted rows on rejoin. See docs/deployment/topology-guide.md.
    • Batching is by BYTE BUDGET as of LocalDb 0.2.0 — LocalDb:Replication:MaxBatchBytes (default 2 MB, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in SyncSession.PumpLoopAsync, with MaxBatchSize demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. The rig's old MaxBatchSize = 16 pin (a hand-computed byte-budget proxy: ~70 KB worst-case config_json x the 500 default is ~35 MB) is retired, but MaxBatchSize is NOT fully redundant with MaxBatchBytes — it also bounds the separate DB READ page in OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materializes the whole page into memory before the byte-budget split runs, so an unset (500-default) MaxBatchSize still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under MaxBatchBytes (arch-review adversarial finding F2). Both site-a nodes on docker/ therefore pin an explicit "MaxBatchSize": 64 to bound that transient allocation, while MaxBatchBytes stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
    • CDC registration is conditional, and both directions self-heal at boot (arch-review WP1.3 + WP3.3, Host/SiteLocalDbSetup.cs). Capture triggers are installed only when this node has replication configured — PeerAddress or ApiKey, an OR because only the dialling half sets PeerAddress while the passive half carries the key alone. An unreplicated node calls DeregisterReplicated on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with baselineExistingRows: true, which seeds __localdb_row_version for pre-existing rows at the LWW floor (HLC 0, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused __localdb_oplog_hlc index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
    • LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to v3 (a new __localdb_snapshot_state table, upgraded in place on open); still wire-compatible, and lib_schema_version deliberately stays 1 because the handshake compares it fail-closed for equality. The extension is negotiated by capability (Handshake.supports_partial_snapshot, SnapshotBegin.tables where empty == all tables), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance last_applied_remote_seq past deltas it never received. For the same reason a partial snapshot moves no watermark on either side; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new SnapshotAck rather than on enqueue, so a crash mid-snapshot re-streams the same scope. SiteLocalDbSetup.cs needed no restructuring — it already calls RegisterReplicated once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: adding an eleventh table to ReplicatedTables on an already-replicating site now snapshots that one table, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
    • LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change): (1) the sync pump now faults the live session when a cap-prune invalidates its position (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public LocalDbOplogGapException through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) transport keepalives both sides — client SocketsHttpHandler pings via new validated LocalDb:Replication:KeepAlivePingDelay/KeepAlivePingTimeout (defaults 60s/20s, Always policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (Host/SiteKestrelSetup.cs — server-wide, also governs the metrics port); (3) the server's single-stream guard is now newest-wins with an awaited handover — a fresh authenticated Sync cancels a dead/wedged incumbent and waits for it to finish (TakeoverTimeout 10s) before starting, so a half-open link can no longer wedge the pair behind AlreadyExists while the zombie side reports LocalDbReplicationConnected=true.
    • LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet): two latent LwwApplier hazards closed before any table could hit them. (1) RegisterReplicated now REFUSES a table that is the parent of a foreign key with a row-mutating ON DELETE action (CASCADE/SET NULL/SET DEFAULT) from any other table (LocalDbRegistrationException naming child.column ON DELETE <action>; RESTRICT/NO ACTION still allowed) — the applier's INSERT OR REPLACE runs with foreign_keys=ON, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero REFERENCES across 29 CREATE TABLEs) — keep it that way; a cascading-FK parent can never be one of ReplicatedTables (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now atomic under a SAVEPOINT — a fault between the two statements used to leave a deleted row with a live ledger entry, which SnapshotStreamer warns-and-skips forever; poison-catch semantics unchanged (narrow SqliteException, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. Both are pinned here (2026-08-20): SiteLocalDbReplicatedTableFkGuardTests asserts at build time that no ReplicatedTables entry has an inbound row-mutating FK (register row 105), and pre-2026-08-20 site files must be VACUUMed once offline to get auto_vacuum=INCREMENTAL — docker/vacuum-site-localdb.sh (pair stopped together; done on all three rig pairs; topology guide → one-off offline VACUUM; row 104). LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only): a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots as_of_seq 0, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at last_acked_seq; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
    • LocalDb 0.2.1 (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) DeregisterReplicated was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an unbounded channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now bounded at 64 messages with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate MaxBatchSize: 64 DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
• All timestamps are UTC throughout the system.
• Inter-cluster communication uses three transports, not two — all cross-cluster command/control and data now rides gRPC after the ClusterClient→gRPC migration's Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md) deleted Akka ClusterClient/ClusterClientReceptionist: (1) gRPC command/control — site→central over the central-hosted CentralControlService (GrpcCentralTransport, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted SiteCommandService (GrpcSiteTransport, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) gRPC server-streaming for real-time data (attribute values, alarm states, SiteStreamService); and (3) plain token-gated HTTP for the deployment config itself — notify-and-fetch, the site pulls the config from DeploymentConfigEndpoints (ManagementService/DeploymentConfigEndpoints.cs) with an X-Deployment-Token header, AllowAnonymous with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (ControlPlaneAuthInterceptor, unchanged). There is no receptionist registration — discovery is by dialling configured endpoints; central builds one SitePairChannelProvider per site (addresses from Site.GrpcNodeAAddress/GrpcNodeBAddress, refreshed from the DB every 60s and on admin changes), sites dial ScadaBridge:Communication:CentralGrpcEndpoints (both central nodes, h2c on CentralGrpcPort 8083, NOT via Traefik). Discovery is asymmetric by design: central discovers site gRPC addresses from the database (refreshable at runtime), sites discover central from appsettings (CentralGrpcEndpoints, static — restart required; StartupValidator requires a Site node to list at least one). Akka.Cluster.Tools stays for ClusterSingleton; only the ClusterClient part is gone. Central never buffers for an unreachable site — the send fails with the caller's Ask/deadline timing out; a ConnectionStateChanged mechanism built for this was deleted as dead code.
• All clusters share ONE ActorSystem name, "scadabridge" — hardcoded in AkkaHostedService at the ActorSystem.Create call. Central and each site are separate clusters only by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has not been re-verified, so treat the name as fixed until someone checks.
• ActiveNodeEvaluator.SelfIsOldestUp is THE single definition of "active node" (Communication/ClusterState/ActiveNodeEvaluator.cs) — the oldest Up member in a role scope, and explicitly never cluster.State.Leader: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence oldest-Up == where ClusterSingletonManager places singletons is the design. ClusterActivityEvaluator.SelfIsOldest, the S&F delivery gate, /health/active and the heartbeat IsActive stamp all delegate here.
• Site nodes carry two Akka roles: the base Site plus a site-specific site-{SiteId} (AkkaHostedService.BuildRoles). Singletons scope to the site-specific role.
• The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted. Akka remoting sets no enable-ssl, no secure cookie, no trusted-selection-paths — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays h2c, but SiteStreamService is no longer open: ControlPlaneAuthInterceptor (Host/ControlPlaneAuthInterceptor.cs) gates /sitestream.SiteStreamService/ — including the PullAuditEvents/PullSiteCalls RPCs that return audit rows — against a per-site preshared key, fail-closed, constant-time compared, alongside the separate LocalDbSyncAuthInterceptor on /localdb_sync.v1.LocalDbSync/ with its own separate key. Site side: ScadaBridge:Communication:GrpcPsk, in production ${secret:SB-GRPC-PSK-<siteId>}, and StartupValidator refuses to boot a site node without it (an unset key would leave the node healthy-looking but serving nothing). Central side: SitePskProvider resolves SB-GRPC-PSK-{siteId} from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with ScadaBridge:Communication:SitePsks:{siteId} as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
• gRPC streaming channel — note the direction is inverted from the data flow: data moves site→central, but each site node hosts the gRPC server (SiteStreamGrpcServer, Kestrel h2c, port 8083, mapped only in the Site branch of Program.cs) and central is the client, dialling in. Central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory, keyed (siteId, endpoint) — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is additive only and field numbers are never reused (AlarmStateUpdate grew 7→23 fields for the native-alarm mirror, then 24 for AckTime). Generated C# is vendored under Communication/SiteStreamGrpc/ with the <Protobuf> include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all] (it always restores the csproj; an active <Protobuf> item must never be committed — it breaks the Docker build). Stream events are BATCHED as of R2 (2026-08-15) — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: batching_supported on the request (InstanceStreamRequest field 3 / SiteStreamRequest field 2) plus a SiteStreamEvent.batch oneof case (field 4) carrying a new SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }. The proto3 default of that flag IS the negotiation, and it is load-bearing — a batch frame reaches a pre-R2 central as EventOneofCase.None, whose ConvertToDomainEvent returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's ForEachEvent unpack handles as the single-event case. Server side is SiteStreamEventBatcher, a per-subscriber pump downstream of StreamRelayActor's bounded DropOldest channel — so it changes framing only and does NOT move the burst ceiling (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. GrpcStreamBatchMaxEvents (100, 1 disables) and GrpcStreamBatchWindow (25 ms, validated strictly under 250 ms = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. Benchmark-proven at target scale 2026-08-18 (the LoadHarness wire path — real Kestrel h2c + real SiteStreamGrpcClient, --wire-probes-per-site): 899,807 events in 12,462 frames (mean 72.2 ev/frame), wire P99 43.4 ms (5.8× inside the 250 ms budget — a 75-event burst never hits the 100 cap, so every batch pays the full 25 ms window: this workload's worst-case latency posture), zero drops/faults; results doc §8. Histogram scadabridge.site.stream.batch_size, recorded only on negotiated streams (ScadaBridgeTelemetry.MeterName is already in the ObservedMeters allowlist). The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit 34a3f4bb) — Kestrel's MaxStreamLifetime/Grpc.AspNetCore.Server max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and IsLive reflects the gap in between.
• Native alarms are a read-only mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — no ack-back, no central tables; state lives in the site's native_alarm_state, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (ISiteAlarmLiveCache) is transient in-memory only — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See Component-DataConnectionLayer.md / Component-CentralUI.md for the model and the authoring surface.
  • AckTime mirror enrichment + the Alarms script accessor (MES alarm-status API Phase 1, 2026-08-01). AlarmStateChanged carries an additive AckTime (DateTimeOffset?), mirrored on the vendored AlarmStateUpdate proto as field 24 and persisted inside native_alarm_state's metadata_json — deliberately NOT a new column, because that table is RegisterReplicated and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause index 18 = AckedState/TransitionTime) and its observation time of the ack transition for MxGateway, which supplies none. Site Call scripts read alarms via the new Alarms.CurrentAsync() accessor (ScriptRuntimeContext + ScriptGlobals, local Ask on GetAlarmSnapshotRequest, returns Commons.Types.Scripts.ScriptAlarm), mirrored on ScriptCompileSurface AND the Central UI SandboxScriptHost editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: docs/plans/2026-06-30-mes-alarm-status-api.md (Phases 2–4 are deployed config, not repo).
• OPC UA cert trust is site-local and not persisted centrally (follow-up): the verify-endpoint probe captures an untrusted server cert but NEVER trusts it, and DeploymentManager broadcasts TrustServerCertCommand/RemoveServerCertCommand to BOTH site nodes — CertStoreActor runs on every site node, not as a singleton, so PKI stores stay consistent across failover.

External Integrations

• External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
• Dual call modes: ExternalSystem.Call() (synchronous) and ExternalSystem.CachedCall() (store-and-forward on transient failure).
• Error classification: HTTP 5xx/408/429/connection errors = transient; other 4xx = permanent (returned to script).
• Notification Service: SMTP with OAuth2 Client Credentials (Microsoft 365) or Basic Auth. BCC delivery, plain text.
• Email delivery has two transports, selected per config row by SmtpConfiguration.Transport (null/Smtp = default): SMTP (MailKit, Basic/OAuth2) or on-prem Exchange EWS (no-SDK CreateItem SOAP over HttpClient, Basic-over-HTTPS only — https enforced at write gate, adapter AND sender — BCC-only recipients, SendOnly so no Sent-Items copy). Under Ews, Host is the full EWS URL, Credentials is username:password, and Port/TlsMode/OAuth2*/MaxConcurrentConnections are unused. Site-facing behavior is unchanged (delivery stays central-only). Q12 (M365 OAuth2 tenant) is closed as superseded — the OAuth2 SMTP path stays config-selectable but untested against a live tenant. Design: docs/plans/2026-08-10-ews-email-transport-design.md.
• Notification delivery is central-only: sites store-and-forward notifications to the central cluster (target = central, not SMTP); sites never talk to SMTP. Notification lists and SMTP config are no longer deployed to sites; recipient resolution happens at central, at delivery time.
• Notification lists carry a Type discriminator (Email and Sms). Notify.To("list") is type-agnostic; delivery is via per-type INotificationDeliveryAdapter (Email via SMTP; Sms via Twilio REST — SmsNotificationDeliveryAdapter, no SDK, one POST per recipient, per-recipient rollup). List Type is fixed after creation.
• Notify.Send is async and enqueue-only — it buffers the notification into the local SQLite S&F store and returns a NotificationId (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (deferToSweep: true buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with maxRetries: 0 (the "no limit" escape hatch), so notifications retry until central acks and are never parked for retry exhaustion — only a corrupt payload parks them (arch-review 02, Tasks 13/14). Notify.Status(notificationId) returns a status record (status, retry count, last error, key timestamps); answered site-locally as Forwarding while still in the site S&F buffer, otherwise round-trips to central.
• SMS delivery adapter (T9/T10, 2026-06-19): SmsNotificationDeliveryAdapter — Twilio REST, no SDK, HTTP Basic auth (AccountSid:AuthToken), one POST .../Messages.json per recipient. SmsConfiguration entity (AccountSid, AuthToken encrypted via Data Protection, FromNumber/MessagingServiceSid, ApiBaseUrl, timeout/retry) stored centrally; managed Admin-only via CLI notification sms list|update + Central UI /notifications/sms. Central-only (never deployed to sites). Per-recipient rollup: all-accepted → Delivered; any-transient → retry/park; mix → delivered-to-good + note; all-permanent → Park. SmsConfiguration (Auth Token in SecretsBlock) travels in Transport bundles via --sms-configs.
• Inbound API: POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object, List).
• Inbound API on a STANDBY central node RELAYS to the active peer (2026-08-28) instead of answering 503 STANDBY_NODE. Method scripts/Route.To() still run only on the active node (IActiveNodeGate); the standby's InboundApiEndpointFilter hands the request to IStandbyRequestForwarder (HttpStandbyRequestForwarder, plain HTTP, verbatim method/path/query/body/headers, peer response replayed unchanged) when ScadaBridge:InboundApi:StandbyForwardPeerEndpoints lists the other node's HTTP base URL — a static appsettings list, because the Akka member address carries the remoting port, not the HTTP one. Empty list = old 503 behaviour. Peers are tried in order; a peer 503 stamped X-ScadaBridge-Standby: 1, a connection failure or a StandbyForwardTimeout (60 s, must exceed the longest method timeout) expiry each move to the next, all failing → 503 ACTIVE_NODE_UNREACHABLE. One-hop loop guard: relayed requests carry X-ScadaBridge-Forwarded and are never relayed again. The 413 body cap runs before the relay, and the standby's AuditWriteMiddleware skips its own row for a relayed call (InboundApiEndpointFilter.RelayedToActiveItemKey) so it is audited once, on the node that executed it. Both rigs (docker/, docker-env2/) configure it on both central nodes; QA/prod appsettings are user-held and need the key added by hand. Spec: Component-InboundAPI.md → Standby forwarding.
• DelmiaNotifier rig stand-in + the per-node inbound-key-store trap (2026-09-02). docker/seed-delmia-sim.sh (in-memory SimMachine composing DelmiaReceiverSim as DelmiaReceiver, a ValueChange script on the ROOT template playing the Galaxy side) + seed-delmia-api.sh (prod's DelmiaRecipeDownload method + a scoped key) + verify-delmia-notifier.sh (notifier A–G incl. an active-central docker kill, takeover measured 27 s, rejoin as standby). The site script is a static-attribute port of the prod handshake — WriteBatchAndWaitAsync rejects static attributes, so it uses per-attribute SetAsync + WaitAsync. Two facts the drill established: (1) docker stop is a graceful CoordinatedShutdown handover (takeover 0 s) — crash drills must docker kill; (2) inbound API keys live in a PER-NODE SQLite file (ScadaBridge:InboundApi:ApiKeyStore:SqlitePath, each central's own volume), written by whichever node serves the management call, so a key exists on one node only and a central failover makes every inbound client 401 — the standby-relay path only works because the standby forwards before authenticating — resolved by register row 108 / the shared store; sync-inbound-api-keys.sh is deleted. Results: docs/api-specs/Delmia-Recipe-Download-API.md §8.

Templates & Deployment

• Pre-deployment validation includes semantic checks (call targets, argument types, trigger operand types).
• Composed member addressing uses path-qualified canonical names: [ModuleInstanceName].[MemberName].
• Override granularity defined per entity type and per field.
• Template graph acyclicity enforced on save.
• Flattened configs include a revision hash for staleness detection.
• Deployment identity: unique deployment ID + revision hash for idempotency.
• Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
• Bulk site deployment (DeploymentService.DeploySiteAsync, arch-review WP2.5) redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (SiteDeploymentMaxParallelism, default 4), serial Finalize — still not all-or-nothing: any one instance's failure doesn't block the rest. Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2) — they are reported in the summary's additive SkippedInstances list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6) — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so DeploySiteAsync pre-filters through IStaleInstanceProbe (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in SkippedInstances with reason up-to-date; the filter is conservative (every uncertainty deploys: missing snapshot, ShouldQuerySiteBeforeRedeploy, null/faulting probe). Additive MgmtDeploySiteCommand.Force / CLI deploy site --force restores the unconditional push for what central cannot see (restored volume, rebuilt node) — Disabled still wins under --force. Also P7-5: the three phases share one scoped DbContext and a failed SaveChanges leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records InProgress; MarkDeploymentFailedAsync now detaches pending changes and retries the Failed write once on the clean tracker; Prepare's per-instance reconcile also runs under a short SiteDeploymentReconcileTimeout (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The PendingDeployment staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
• Create Instance picks its template from the Templates folder tree (2026-08-21): the flat <select> is gone. The Template field is a read-only folder-qualified path box (TemplatePathFormatter, "Folder / Sub / Name") + a Choose… button, both opening TemplatePickerDialog — an IDialogService.ShowAsync body wrapping the same shared TemplateFolderTree the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive RevealKeys parameter (ancestors of the named f:/t: keys start expanded when no Filter is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). InstanceCreate gained [SupplyParameterFromQuery] TemplateId, which the Templates tree's new Deployer-gated "New Instance…" context-menu item navigates with; combined with ?siteId= it populates the Connection Bindings card on first render. The trap for anyone extending the picker: in TreeViewSelectionMode.Single the inner TreeView reads the singular SelectedKey/SelectedKeyChanged pair, but TemplateFolderTree forwards only the plural (Checkbox-mode) SelectedKeys pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own NodeContent, as the picker does. Design: docs/plans/2026-08-21-template-picker-design.md.
• Create Instance binds module groups at creation (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive IFlatteningPipeline.PreviewAsync (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses BindingGroupBuilder; NodeBrowserDialog.SelectableNodeClasses lets it pick Object nodes (default stays Variable-only for every existing caller). Create = CreateInstanceAsync → SetConnectionBindingsAsync (the existing upsert, no new server surface) → always navigate to Configure with ?created=1&bound=N|&bindError=…, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: docs/plans/2026-08-21-create-instance-bindings-design.md.
• Site-side apply is all-or-nothing per instance.
• System-wide artifact version skew across sites is supported.
• Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
• Optimistic concurrency on deployment status records.
• Naming collisions in composed feature modules are design-time errors.
• Transport (#24, M8): bundles now move site/instance config (Sites, site-scoped DataConnections, Instances + override children + Area by name), not just central-only config — but still never touch site runtime nodes (imported instances land NotDeployed). A BundleNameMap name-mapping subsystem (Commons: SiteMapping/ConnectionMapping/MappingAction) reconciles environment-specific identifiers (sites by SiteIdentifier, connections by site-qualified {SiteIdentifier}/{Name}, instances by UniqueName): auto-match → operator override via the import-wizard Map step / CLI --map-site/--map-connection/--create-missing-* → blocker rows for the unresolved. D3 "carry full config (encrypted secrets)": Site addresses travel; DataConnection config rides the encrypted SecretsBlock (presence-only in diffs). Per-line Myers diff for code fields via the pure LineDiffer (ArtifactDiff embeds a size-capped structured lineDiff; input capped at MaxInputLines=4000 → summary-only, so a crafted bundle can't OOM the active node); ImportResult.StaleInstanceIds is real via the IStaleInstanceProbe seam (Commons; implemented in DeploymentManager). schemaVersion 1.0→1.1 (additive); bundleFormatVersion stays 1; no new EF tables/columns. Arch-review 05 (plan 05): template child fidelity completed (round-trip-guard-verified: LockedInDerived, native-alarm-sources, script cadence/timeout, real AreaName, ES retry config + methods-on-Add, notification recipients, folder parent edges); import runs the script trust gate (template/shared/ApiMethod bodies, template script + alarm Expression-trigger bodies, and instance alarm-override trigger expressions — R2-05) as a hard pre-runtime gate + a name-resolution severity split (template=advisory ConflictKind.Warning+ImportResult.Warnings, ApiMethod=hard error); ArtifactDiff resolves folder/base/composition names (unchanged structured templates classify Identical again); MaxConcurrentImportSessions=8 session cap (startup-validated >0, R2-05). R2-05 also: the post-commit ScriptArtifactsChanged publisher now covers Add resolutions (not just Overwrite/Rename), import warns on instance overrides targeting locked template members (advisory, rows still written), and the Expression-trigger syntax check is surface-keyed-cached + skipped on read-only staleness/comparison paths (validateScriptCompilation:false). Rename does NOT rewrite call sites (deploy gate on the target is the real check). Plan-06 handoff: ScriptArtifactsChanged/IScriptArtifactChangeBus seam + Host in-process bus + Transport post-commit publisher shipped (ApiMethod consumer = InboundAPI ScriptArtifactChangeSubscriber, wired in plan R2-06; SharedScript/Template publish with no consumer — nothing at central caches those kinds).

Store-and-Forward

• Fixed retry interval, no max buffer size. Only transient failures buffered.
• Async best-effort replication to standby (no ack wait).
• Messages not cleared on instance deletion.
• CachedCall idempotency is the caller's responsibility.
• Notification Outbox: central NotificationOutboxActor singleton on the active central node — the first centrally-hosted store-and-forward component (the S&F Engine remains site-only).
• Notifications table in central MS SQL is the single source of audit truth (one row per notification); type-agnostic via the Type discriminator.
• Status lifecycle Pending → Retrying → Delivered / Parked / Discarded, plus site-local Forwarding (never persisted centrally).
• Dispatcher loop polls due rows, resolves the list, delivers via the typed adapter; transient failures retry to Parked, permanent failures park immediately.
• Site→central handoff is at-least-once: ack-after-persist plus insert-if-not-exists on NotificationId.
• No Akka replication — MS SQL is the HA store; daily purge of terminal rows after a configurable window (default 365 days).
• Notification Outbox retry reuses central SMTP max-retry-count and fixed interval for Email; Sms uses SmsConfiguration.MaxRetries/RetryDelay, falling back to the SMTP policy when unset.
• Cached calls (ExternalSystem.CachedCall, Database.CachedWrite) return a TrackedOperationId tracking handle, unified with Notify.Send's existing tracking model (Notify.Status retained as a thin alias).
• A site-local operation tracking table (SQLite, alongside the S&F buffer) is the source of truth for cached-call status; Tracking.Status(id) reads it site-locally and authoritatively; terminal rows purged after a configurable window (default 7 days). Cached-call telemetry is OFF the script thread as of 2026-09-06 — a Flow Debugger run showed a Database.CachedWrite spending ~160 of a 176 ms site run on bookkeeping. The real cost was NOT the SQLite commits: CachedCallTelemetryForwarder awaited IAuditWriter.WriteAsync three times, and SqliteAuditWriter.WriteAsync returns the batch completion, which lands only after the writer's FlushIntervalMs (50 ms) coalescing wait — so never await an audit write on a script path; enqueue and observe the fault on a continuation, as every other emitter already did. Same change made OperationTrackingStore a single-writer queue (FIFO, batch-per-transaction pump, reads barrier on earlier queued writes so Tracking.Status keeps read-your-writes, DisposeAsync drains, sync Dispose() doesn't; trade: a queued-but-unapplied row is lost on a hard crash, ms window). Measured: inbound 194 → 25 ms, script run 176 → 17 ms. Design + numbers: docs/plans/2026-09-06-tracking-store-async-writes-design.md.
• Unified tracking status lifecycle Pending → Retrying → Delivered / Parked / Failed / Discarded; Failed = permanent failure (also returned synchronously to the calling script). No Forwarding state for cached calls.
• Site Call Audit (#22): central SiteCallAuditActor singleton with a SiteCalls audit table (central MS SQL) fed by best-effort site telemetry plus periodic reconciliation pulls — an eventually-consistent mirror, NOT a dispatcher; cached-call delivery stays site-local. Ingest is insert-if-not-exists then upsert-on-newer-status.
• Central UI Site Calls page + central→site RetryParkedOperation/DiscardParkedOperation relay for parked cached calls; central never mutates the SiteCalls row directly.

Centralized Audit Log

• Layered design — append-only AuditLog (#23) sits alongside operational Notifications (#21) and SiteCalls (#22), not replacing them.
• Scope = script trust boundary: outbound API (sync + cached), outbound DB (sync + cached), notifications, inbound API. Framework/internal traffic is explicitly excluded.
• One row per lifecycle event; cached calls produce 4+ rows per operation (Submitted, Forwarded, Attempted, Delivered/Parked/Discarded).
• ExecutionId (uniqueidentifier NULL) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; CorrelationId remains the per-operation lifecycle id (NULL for sync one-shots).
• ParentExecutionId (uniqueidentifier NULL) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's ExecutionId; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested CallScript/CallShared invocations; IX_AuditLog_ParentExecution backs the filter + the recursive execution-tree walk. Tag-cascade (alarm leg) is populated, not just plumbed — M5.4 T4 threaded the parentExecutionId parameter but every AlarmActor.SpawnAlarmExecution call site passed null, so alarm runs were silently always roots; the id now rides site-locally as SetStaticAttributeCommand.SourceExecutionId → AttributeValueChanged.SourceExecutionId → SpawnAlarmExecution (all additive/nullable, no wire/proto/schema change; Expression triggers capture the writer with the evaluated snapshot since the eval completes off-dispatcher). Sources are ScriptRuntimeContext.SetAttribute (the run's own ExecutionId) and Route.To(...).SetAttributes(...) (the inbound request's). Still roots by design, not omission: alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a data-sourced attribute, so only static writes cascade), and ScriptActor value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval run has no single identifiable write).
• Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on EventId; periodic reconciliation pull as fallback when telemetry is lost. The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables — at the required AuditLog:SiteWriter:DatabasePath (arch-review WP1.2, finding #2 High), /app/data/auditlog.db on the rig; StartupValidator refuses to boot a Site node without it. DatabasePath used to default to CWD-relative auditlog.db, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors FlushIntervalMs (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets PRAGMA synchronous = NORMAL (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
• Cached operations: site emits a single additively-extended CachedCallTelemetry packet carrying both audit events and operational state; central writes AuditLog + SiteCalls in one transaction.
• Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to InboundMaxBytes (default 1 MiB); request headers stored in Extra.requestHeaders (post-redaction); per-method SkipBodyCapture flag suppresses bodies while still recording headers + metadata; AuditInboundCeilingHits counter surfaced on health snapshot. (M5.3 T7)
• Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
• 365-day central retention with monthly partition-switch purge; per-channel retention overrides (AuditLog:PerChannelRetentionDays) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard ForwardState invariant (no row purged until forwarded or reconciled).
• Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE); hash-chain tamper evidence (T1) and Parquet archival (T2) are deferred to v1.x — not shipped in M5.
• Node-of-origin is captured alongside site-of-origin: SourceNode (varchar(64) NULL) on AuditLog, Notifications, and SiteCalls — node-a/node-b for site rows (qualified by SourceSiteId/SourceSite), central-a/central-b for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc) on AuditLog.
• Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose PerNodeNotificationKpiRequest/PerNodeSiteCallKpiRequest messages that group stuck/parked/delivered counts by SourceNode, surfacing per-node breakdowns on the Health dashboard.
• audit tree --execution-id <guid> CLI command (M5.3 T8) + GET /api/audit/tree endpoint — resolves any node to its chain root and renders the full execution tree; backed by IAuditLogRepository.GetExecutionTreeAsync.
• Central UI: new top-level Audit nav group + Audit Log page, with drill-ins from Notifications, Site Calls, External Systems, Inbound API Keys, Sites, and Instances.

Security & Auth

• Authentication: direct LDAP bind (username/password), no Kerberos/NTLM. LDAPS/StartTLS required.
• Cookie+JWT hybrid sessions: HttpOnly/Secure cookie carries an embedded JWT (HMAC-SHA256 shared symmetric key), 15-minute expiry with sliding refresh, 30-minute idle timeout. Cookies are the correct transport for Blazor Server (SignalR circuits).
• LDAP failure: new logins fail; active sessions continue with current roles.
• Load balancer in front of central UI; cookie-embedded JWT + shared Data Protection keys for failover transparency.
• Two-person MxGateway secured writes (M7): two new global roles — Operator (initiates) + Verifier (approves) — added alongside the canonical Administrator/Designer/Deployer/Viewer, with RequireOperator/RequireVerifier policies. An Operator submits a secured write from the Central UI Secured Writes page (/operations/secured-writes); it stays a Pending PendingSecuredWrite row until a distinct Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a WriteTagRequest to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort AuditChannel.SecuredWrite / AuditKind.SecuredWrite* central-direct-write row sharing the row id as CorrelationId. (SecuredWrite audit rows stamp SourceNode via ICentralAuditWriter/INodeIdentityProvider.) Pending secured writes expire server-side after a configurable TTL (ManagementServiceOptions.SecuredWritePendingTtl, default 24 h): an overdue Pending row is CAS'd to Expired (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, AuditKind.SecuredWriteExpire).
• Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108). The shared ZB.MOM.WW.Auth.ApiKeys library stores inbound keys in SQLite per node unless ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer, which points both central nodes at the same zbauth schema in the configuration database (library ZB.MOM.WW.Auth.ApiKeys.SqlServer 0.3.0) — under SqlServer, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed *.imported. Central pairs MUST run Provider=SqlServer — a clustered Central left on Sqlite boots fine but StartupValidator logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (docker/, docker-env2/) run SqlServer; QA (vq03/vq04) and prod (vd03) appsettings are user-held and their rollout is a deployment record, not yet applied. Design: docs/plans/2026-09-02-inbound-api-key-shared-store-design.md; runbook: docs/operations/inbound-api-key-reissue.md → Central pairs.

Cluster & Failover

• auto-down downing strategy (decision 2026-07-21 — availability over partition-safety). Akka's AutoDowning provider, auto-down-unreachable-after = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. keep-oldest remains a supported SplitBrainResolverStrategy value (partition-safe, but an oldest-crash is a total outage — Akka's down-if-alone only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: docs/plans/2026-07-21-auto-down-availability-decision.md. Ignore the boot-time Akka warning claiming "keep-majority will be used instead" — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit AutoDowning provider-class BuildHocon sets. But AutoDowning IS deprecated, so every Akka.NET version bump must re-run DRILL_MODE=active bash docker/failover-drill.sh (register row 70).
• Both nodes are seed nodes. min-nr-of-members = 1.
• Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — docker/failover-drill.sh).
• CoordinatedShutdown for graceful singleton handover.
• Automatic dual-node recovery from persistent storage.
• Active/standby is decided by ActiveNodeEvaluator.SelfIsOldestUp, never by cluster leadership — see the Architecture note above. Central's /health/active backs both Traefik's active-node routing and IActiveNodeGate, so the proxy and the Inbound API always agree on which node is active. Site nodes DO map /health/* too — the "site nodes map no /health/* at all" claim that stood here was stale: SiteServiceRegistration registers akka-cluster + localdb (Ready) and SitePairActiveNodeHealthCheck (Active, role-scoped to site-{SiteId}, deliberately NOT central's unscoped shared check), and Program.cs's Site branch calls MapZbHealth() on the HTTP/1.1 listener so the family overview dashboard can probe every node identically. Nothing in this repo probes a site's /health/active (Traefik health-probes central only), and there is no IHealthCheckPublisher anywhere, so those checks run only on an inbound HTTP probe. Consequence worth knowing: the standby's correct-by-design active-node Unhealthy verdict is logged by DefaultHealthCheckService at Error on every probe, which LoggerConfigurationFactory now suppresses with one narrowly-keyed Serilog filter (register row 71 — see Component-Host.md → Standby health-check log noise; genuine failures of every other check still log at Error). Central reaches a site by dialling GrpcNodeAAddress/GrpcNodeBAddress explicitly and flipping on error; whichever node answers, the site-internal ClusterSingletonProxy lands the work on the active node for free, so central still never needs to track which site node is active.
• Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED. Only seed-nodes[0] may self-join to form a new cluster (Akka runs FirstSeedNodeProcess for it, JoinSeedNodeProcess — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, seed-node-timeout); StartupValidator fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the InitJoin handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. An external self-form timer (Cluster.Join(SelfAddress) after a window) was implemented and REJECTED: it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: SelfFirstSeedBootstrapTests. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See docs/requirements/Component-ClusterInfrastructure.md → Seed Node Ordering.
• Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed. Self-first-on-both means a truly simultaneous cold start (shared power/hypervisor event) races FirstSeedNodeProcess on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (lmxopcua d1dac87f): a dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default OFF, guard-off byte-identical). On: BuildHocon emits an EMPTY seed list (Akka does not auto-join) and ClusterBootstrapCoordinator (IHostedService, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated JoinSeedNodes — the pure ClusterBootstrapGuard core makes the lower canonical host:port the founder (self-first, forms immediately), the higher node TCP-probes the founder up to PartnerProbeSeconds (25s) and joins peer-first if reachable else self-first (cold-start-alone preserved). Decided BEFORE a single join, never re-forms mid-handshake (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated >0 when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: ClusterBootstrapGuardTests (pure) + ClusterBootstrapCoordinatorTests (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). ENABLED on the docker rig (all 8 nodes) 2026-08-02 after deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology docker compose restart), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See docs/requirements/Component-ClusterInfrastructure.md → Simultaneous cold start.

UI & Monitoring

• Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
• UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the frontend-design skill when designing UI pages/components.
• Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via the SiteCommandService query surface — QueryReply.DebugViewSnapshot). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
• Flow Debugger (2026-09-05, Deployment → /deployment/flow-debugger + CLI flow-debug run-method|run-script|run-value-change|replay). Runs one entry point (an inbound API method, or a site script) and draws every step as a Central / Site / Tags / External swimlane; Live really runs it, Replay rebuilds a past execution. v1.1 (2026-09-06, docs/plans/2026-09-06-flow-debugger-outbound-calls.md): every ExternalSystem.* / Database.* / Notify.Send a traced run makes is an ApiCall / DbCall / Notify node on the External lane — the site routes those already-existing ApiOutbound / DbOutbound / Notification rows through the same inline trace collector, the replay query widens to those channels, and FlowGraphBuilder folds a cached operation's rows by (kind, CorrelationId) into ONE node with a Lifecycle (sync calls in-chain, cached/notify on dashed Async edges, trigger cascades on Spawn edges, the Response linked from the inbound execution's last step). Later lifecycle rows are not inline, so after a Live run the page polls the Replay command (5 s, ≤ 2 min) while any node is Pending, plus a Refresh outcomes button. A refresh MERGES, never replaces (2026-09-06, found live in the browser): the page unions the re-read rows into the run's own rows by EventId and rebuilds the graph, because a Live run carries its site rows inline while the same rows reach central's audit store by telemetry ~5–30 s later — a Replay read taken in between lacks them, and the old wholesale swap made an already-drawn node (a resolved cached DB write) vanish until the next manual refresh. The inline rows ARE the audited rows (FlowTraceCollector sinks the same AuditEvent objects the site writer persists), so the union is exact. Central method-script Database.* calls (InboundDatabaseHelper) are now audited (DbOutbound/DbWrite) — they were not. Rig demo: docker/seed-flow-outbound-demo.sh + verify-delmia-notifier.sh step I. Trace steps ARE audit events — AuditChannel.ScriptTrace with kinds RouteCall (central, always-on per Route.To().Call, joins the two halves of a Replay) and ScriptRun / AttributeWrite / AttributeWait (site, gated by a bool Trace riding beside ParentExecutionId on RouteToCallRequest / ScriptCallRequest / SetStaticAttributeCommand / AttributeValueChanged; proto fields additive, site_command.proto imports sitestream.proto). A Live run gets the site rows inline on RouteToCallResponse.TraceEvents (+ ChildExecutionId) because the 5 s telemetry cadence is too slow; Replay reads them from AuditLog (GetTraceEventsAsync). A value-change on a data-sourced attribute is a REAL tag write; the Instance Actor's trace arm (SiteRuntime:TraceArmWindowSeconds, 30) stamps the device echo with the writer's execution id so the triggered run is parented and traced, and central polls the site's new PullTraceEvents RPC (point read, not reconciliation) for it. Execution path is four management commands on the per-node ManagementActor (FlowDebug*; the three run commands are Deployer-only, Replay is wider) — a standby refuses the three run commands with STANDBY_NODE (shown verbatim); ApiMethod.TraceAllCalls (migration AddApiMethodTraceAllCalls, CLI --trace-all-calls) traces every real call to a method for later Replay (7–10 rows/call). Prediction before a run is a Roslyn syntax walk (FlowShapePredictor, REQ-SA-7, DB-free); the graph (FlowStep/FlowGraphBuilder, Commons) is a central projection, never a wire type; the renderer is custom SVG (FlowGraphView). Rig: AuditLog:PerChannelRetentionDays:ScriptTrace=30 on both centrals; verify-delmia-notifier.sh step H proves live + replay. Mixed-version caveat: an older site ignores the flag → central RouteCall only, page says site not traced. v1.2 (2026-09-06, docs/plans/2026-09-06-flow-debugger-followups.md): a Parked External node offers Retry / Discard in place (same site-scope check, confirm and relay as the report pages, via IFlowDebugService.RetryOutboundAsync/DiscardOutboundAsync, then the page re-polls Replay until the node leaves Parked) and every folded node deep-links to /site-calls/report?correlationId= / /notifications/report?correlationId= (both pages accept it through additive TrackedOperationIdFilter / NotificationIdFilter query params and auto-open the row); register row 111 closed. Trap closed the same day: FlowDebugService.PredictMethodAsync used to resolve route targets with GetAwaiter().GetResult() inside the synchronous predictor walk — on the Blazor renderer sync context that DEADLOCKED the whole circuit (page inert, healthy websocket, no error) whenever a method routing to a site script was chosen; now a resolve-and-rewalk loop, pinned by a single-thread-pump test. Never block on a task from a Blazor event/lifecycle path. Browser checks of the page go through Playwright (FlowDebuggerTests, fixture login + an ephemeral method), not the login-disabled override. Arch-reviewed 2026-09-06 (docs/plans/2026-09-06-flow-debugger-arch-review.md, remediation docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md): the inline Live trace copy is now redacted and capped by the shared FlowTraceCollector (500 events and 1 MB of DetailsJson, F1), a method-entry Live run enforces the operator's site scope at route time (RouteHelper.WithSiteScope → SITE_SCOPE_DENIED → ManagementUnauthorized, F2), and the page/Replay opened to the OperationalAudit roles under the new RequireFlowDebugger policy while everything that executes stays Deployer-only (F3). Design docs/plans/2026-09-05-flow-debugger-design.md; plan docs/plans/2026-09-05-flow-debugger.md. v1.3 (2026-09-08, docs/plans/2026-09-08-flow-debugger-script-entry-design.md): the second entry point is site → instance → script; ValueChange/Conditional scripts are fired by writing the trigger's attribute (derived from the trigger config, not picked), every other kind is a direct traced RouteToCall with a ParameterValueForm from the script's ParameterDefinitions and one operator-attributed RouteCall root row (FlowDebugRunScriptCommand, Deployer-only, long-running); run-value-change stays as the attribute-level CLI escape hatch (fires Expression triggers). Instance picker (2026-09-08, docs/plans/2026-09-08-instance-picker-design.md): Flow Debugger and Debug View both replaced their flat instance <select> with the shared InstancePicker combobox — a text field whose dropdown is the site's area tree (InstanceTreeBuilder, all areas collapsed on open), typed text filters and expands to the matches; Areas/Instances are compared by reference, so callers must assign the loaded lists once per site change, never per render, or the panel closes on every re-render. v1.4 (2026-09-09, docs/plans/2026-09-09-flow-debugger-step-logs-design.md): scripts can log — a Log.Debug/Info/Warn/Error(message, exception?) global on all four surfaces (site ScriptGlobals, inbound InboundScriptContext, the ScriptCompileSurface deploy gate as CompileLog, and both Test Run sandboxes/editor hosts, where it appends to the console pane), always reaching the node ILogger and emitting a ScriptTrace/ScriptLog audit row (one per line) only when the run is traced, so an untraced timer script costs no audit volume. FlowGraphBuilder folds those rows onto the step that produced them (FlowStep.Logs) and never draws them as nodes — attribution is Extra.stepEventId, pre-minted by SetAttribute/WriteBatchAndWait/WaitInternal/sync ExternalSystem.Call through an AsyncLocal TraceStepScope, else the execution's anchor row, else its first step, else FlowGraph.UnattachedLogLines; the runtime's own Information+ entries are captured for free by a TraceCapturingLogger decorator instead of editing every _logger.LogWarning call site. The failing step now carries the whole exception (errorDetail = ex.ToString() on the site ScriptRun row and on the central inbound row — the latter never on the HTTP body, which stays SCRIPT_ERROR). The two halves are NOT gated alike: the central inbound row's errorDetail is stamped on every failing method request, traced or not (EndpointExtensions stashes it on HttpContext.Items, AuditWriteMiddleware consumes it), bounded only by the redactor's ApiInbound cap AuditLog:InboundMaxBytes (1 MiB default; the 64 KB ErrorCapBytes is what bounds every OTHER channel's error rows, the site ScriptRun row included) — tracing only decides whether the row is later readable as a Flow Debugger step; the site ScriptRun errorDetail is traced-only, because an untraced run emits no ScriptTrace row at all, rendered as a red Type: message subtitle plus a Log (n) tab; the node badge is ✕ n ! n ≡ n and the strip counts Log: n errors · n warnings · n info (a Log.Error is a line, not a failed step — it never moves the state chip). Bounded: ScadaBridge:SiteRuntime:MaxLogLinesPerRun / ScadaBridge:InboundApi:MaxLogLinesPerRun (200, validated > 0), 2 000-char messages, one closing warn/runtime row naming the dropped count, and a low-priority inline lane (FlowTraceCollector.Add(evt, lowPriority: true), log rows stop 50 events short of the count cap and at 75 % of the byte budget, reserving the rest for the step rows written at run end) whose overflow sets LogsTruncated → the page badge inline log lines capped — refresh outcomes pulls the rest, which clears once a refresh merges in more log rows. "Never await an audit write on a script path" still holds — every log row is enqueued fire-and-continue, exactly like the 2026-09-06 cached-call telemetry fix. Trap: build ScriptLogger over the UNDECORATED logger — TraceCapturingLogger refuses to wrap another TraceCapturingLogger (it would emit two rows per line). CLI flow-debug … --show-logs + a logs column; rig demo docker/seed-flow-log-demo.sh + verify-delmia-notifier.sh step J. Mixed-version: an OLD central projects a NEW site's ScriptLog rows as Unknown nodes — upgrade central first, as always. Method pane structured inputs (2026-09-10, Gitea #37, docs/plans/2026-09-10-flow-debugger-method-params-design.md): the Run inbound API method pane now renders the same ParameterValueForm as the script pane (keyed on the method id) plus a Form | Raw JSON whole-payload escape hatch (Monaco, schema-validated on every edit); the page keeps ONE value tree and serializes it as-is, the old per-field text coercion is gone, and Run is disabled while the form or the raw document is invalid — so a required array-of-object parameter (MesMoveOut WorkOrders) can be entered, and a failed parse is never sent as a string. The per-method localStorage entry stores the value tree and is dropped on restore if it no longer validates (that also retires the pre-#37 flat string entries).
• Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
• Dead letter monitoring as a health metric.
• Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search. The cap measures site_events + its four indexes, NOT the consolidated LocalDb file (register row 81, 2026-08-17): whole-file measurement made the trim unwinnable on a replicated node — each batch freed event pages while adding an equal count of oplog rows + tombstones to the same file, so it stopped 2.5x over a 64 MB cap and a node with a down peer could not enforce the cap at all. dbstat (the exact per-table measurement) is NOT compiled into the bundled SQLitePCLRaw.lib.e_sqlite3 — probed and pinned by SqliteCapabilityTests, so don't reach for it — the footprint is modelled from sampled row width instead (−0.8%…+12.8%, biased high). Consequence: MaxStorageMb no longer bounds the data volume alone; LocalDb bookkeeping is separate overhead bounded by MaxOplogRows/TombstoneRetention and is deliberately not this component's job. Decision: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md.
• Notification Outbox KPIs are central-computed point-in-time from the Notifications table (global + per-source-site): queue depth, stuck count, parked count, delivered-last-interval, oldest-pending age.
• Stuck = Pending/Retrying older than a configurable age threshold (default 10 min) — display-only (KPI count + row badge), no escalation/alerting.
• Headline KPI tiles surface on the Health dashboard; a new Central UI Notification Outbox page offers a queryable list with Retry/Discard actions on parked notifications.
• Site Call Audit KPIs are central-computed point-in-time from the SiteCalls table (global + per-site), mirroring the Notification Outbox KPI shape; tiles surface on the Health dashboard alongside a queryable Central UI Site Calls page with Retry/Discard on parked rows.
• KPI History & Trends (#26, M6): a reusable central KPI-history backbone — supersedes the prior "point-in-time only, no time-series store" stance — backed by a tall/EAV KpiSample table in central MS SQL (no new infra). A KpiHistoryRecorderActor cluster singleton (kpi-history-recorder, not readiness-gated, best-effort with per-source isolation) samples every minute by enumerating DI-registered IKpiSampleSources (each lives with its owner, registered via TryAddEnumerable, reusing existing KPI/aggregator reads); daily purge after RetentionDays (default 90). Querying is IKpiHistoryRepository.GetRawSeriesAsync → KpiSeriesBucketer (last-value-per-bucket) → scoped dual-ctor KpiHistoryQueryService → a reusable custom-SVG KpiTrendChart (no third-party charting lib). Trends ship on four surfaces: Notification Outbox, Site Calls, Audit Log pages + a per-site Health-dashboard panel. KpiHistoryOptions (ScadaBridge:KpiHistory): SampleInterval 60s, RetentionDays 90, PurgeInterval 1d, DefaultMaxSeriesPoints 200; validated. Hourly rollups shipped 2026-07-10 (deferred #22): a second KpiRollupHourly table folded by a third recorder tick (kpi-rollup, RollupInterval 1h, re-folds RollupLookbackHours window via idempotent failover-self-healing upsert + one-shot backfill on start), per-metric gauge-vs-rate fold (KpiMetricAggregationCatalog), and raw-vs-rollup query routing by RollupThresholdHours (168h) unblock long-range 30d/90d trend windows (rollups retained RollupRetentionDays 365 ≥ raw 90, dual daily purge). M6's T9 and T10 were originally deferred. T9/T10 delivered 2026-06-19 as an SMS (Twilio) adapter — Teams was evaluated and dropped (outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only). NotificationType.Sms added; SmsNotificationDeliveryAdapter (Twilio REST, no SDK) registered alongside Email; Central UI Type selector (adapter-gated) + SMS recipient input; CLI --type sms --phones on list create/update; notification sms list|update commands.

Code Organization

• Entity classes are persistence-ignorant POCOs in Commons; EF mappings in Configuration Database.
• Repository interfaces defined in Commons; implementations in Configuration Database.
• Commons namespace hierarchy: Types/, Interfaces/, Entities/, Messages/ with domain area subfolders.
• Message contracts follow additive-only evolution rules for version compatibility.
• Per-component configuration via appsettings.json sections bound to options classes (Options pattern).
• Options classes owned by component projects, not Commons.
• Host readiness gating: /health/ready endpoint, no traffic until operational.
• EF Core migrations: auto-apply in dev, manual SQL scripts for production. Gotcha: dotnet ef migrations add … --no-build scaffolds off the previously-built DLL and can emit an empty migration — build first (drop the --no-build), and delete the empty Migrations/ files if it happens (migrations remove needs a live DB).
• Audit logging absorbed into Configuration Database component (IAuditService).

Akka.NET Conventions

• Tell for hot-path internal communication; Ask reserved for system boundaries.
• Cross-cluster communication is gRPC (per-site PSK-authenticated): site→central CentralControlService, central→site SiteCommandService, plus the SiteStreamService data stream. ClusterClient/ClusterClientReceptionist were removed in the migration's Phase 4 — service discovery is by dialling configured endpoints, not the receptionist. (Akka.Cluster.Tools remains for ClusterSingleton.)
• Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — ScriptTrustPolicy is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to ScriptTrustValidator. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory. Every Roslyn script-compile surface must also attach the shared CachingScriptMetadataResolver.Instance (ScriptAnalysis) via ScriptOptions.WithMetadataResolver — the four today are Site Runtime ScriptCompilationService, Inbound API InboundScriptExecutor, Central UI ScriptAnalysisService, and ScriptAnalysis RoslynScriptCompiler; a new surface that omits it silently reintroduces a per-compile native metadata leak (see Component-ScriptAnalysis.md).
• Application-level correlation IDs on all request/response messages.

Tool Usage

• When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See src/ZB.MOM.WW.ScadaBridge.CLI/README.md for the full command reference.
• CLI HTTP timeouts are honest (arch-review remediation, finding #1 High). ManagementHttpClient used to cap every call at min(30s, caller timeout), silently truncating deploy site's 5-minute BulkDeployTimeout and the 5-minute bundle export/preview/import calls — the CLI printed a fake 504 Request timed out while the server kept working to completion. HttpClient.Timeout is now Timeout.InfiniteTimeSpan; the per-call CancellationTokenSource is the single overall deadline (connect phase included), with connect bounded separately via SocketsHttpHandler.ConnectTimeout (env override renamed SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its narrower new meaning).

CLI Quick Reference (Docker / OrbStack)

Management URL, test credentials, rebuild/redeploy commands and the infra services are in the scadabridge-cluster-ops skill (.claude/skills/scadabridge-cluster-ops/SKILL.md).