Owner decision on #107 (2026-11-13): enabling AuditLog:HashChain must not
chain the historical backlog. main chained every unchained row oldest-first.
- The first chaining pass on a pristine database (no head, no chained row)
writes a chain-start anchor: AuditLogChainTombstone (cutoff month,
ChainSeq 0), PurgedAtUtc = DB SYSUTCDATETIME(), RowHash =
SHA-256("ScadaBridge.AuditLog.ChainCutoff.v1" || YYYY-MM || ticks).
Written once under the exclusive chain lock, so both centrals agree
whatever their clocks/config. No migration: existing table + grants.
- Only rows with OccurredAtUtc >= cutoff are chained. The anchor hash is
the cutoff month's genesis, binding the cutoff into the chain.
- AuditLog:HashChain:ChainBacklog (default false) restores chain-everything,
read only on the very first enablement. Legacy chains (heads, no anchor)
keep chaining everything.
- verify-chain splits unchained rows into UnchainedRows vs PreCutoffRows
(+ ChainFromUtc); new breaks ChainedBeforeCutoff (row re-dated before the
cutoff, or cutoff moved forward) and CutoffAnchorInvalid (anchor hash /
month / multiplicity). Archive verify uses the anchor from its own
tombstone file or the DB.
- Partition switch-out keeps the anchor (deletes ChainSeq > 0 only); the
head store / export / verify-heads tombstone count excludes it.
- CLI verify-chain table prints Chain from + Pre-cutoff rows.
Tests: AuditLogHashChainCutoffTests (7, MSSQL), 6 verifier unit tests,
service/CLI/management wire tests; 9 mutations all killed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.
- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
the first sample after any (re)connect, triggers a probe of the other endpoint
(short-lived session: connect, read, dispose). Switch only when it reports a
Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
the previous ServiceLevel switch. Both degraded, other unreachable or level
unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
(enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
migration. Central UI: fields in the Backup card next to Failover Retry Count.
Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every CLI option that can carry a secret gets a `--<name>-file <path|->`
sibling ('-' reads stdin), so the value stays out of the process list and
shell history: --password-file (global), --primary-config-file /
--backup-config-file (data-connection create|update), --auth-config-file
(external-system create|update), --connection-string-file (db-connection
create|update), --credentials-file (notification smtp create|update) and
--auth-token-file (notification sms create|update). Follows the existing
--config-file convention; `@path` is avoided because System.CommandLine
expands @-tokens as response files.
The value is read at parse time; a missing/unreadable/empty source, or '-'
on a terminal stdin, is a parse error naming option + path only, never the
content. Inline and file forms are mutually exclusive, at most one option
may read stdin, and one trailing line break is stripped. Inline forms keep
working.
docker/seed-sites.sh and docker-env2/seed-sites.sh now write the MxGateway
config (windev gateway key) to a mode-600 temp file removed by an EXIT trap
and pass --primary-config-file; seed-flow-outbound-demo.sh pipes its rig
connection string via --connection-string-file -. CLI README (new "Secrets
from a file or stdin" section + per-command tables), Component-CLI.md and
both docker READMEs updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.
The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.
Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.
Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OPCFoundation.NetStandard.Opc.Ua.Client and the test-only .Server move
1.5.378.182 -> 2.0.1, the version OtOpcUa master runs (lmxopcua#782; #799
and #801 closed). Transitive Core/Core.Types/Types/Configuration/
Security.Certificates resolve to 2.0.1.
API breaks fixed in the DCL adapter:
- Built-in types are immutable structs (Variant, DataValue, NodeId,
LocalizedText, StatusCode, ArrayOf<T>). New OpcUaValueBoundary restores
the SDK 1.5 CLR shapes for every read, subscription and alarm-event
value (byte[], UTC DateTime with no-timestamp = MinValue, Guid,
detached arrays) and builds write variants with SDK 1.5's
new Variant(object) semantics (byte[] = ByteString scalar).
- StatusCodes members are no longer constants: switches/patterns over
them (verify-outcome mapping, connection-level classification,
BadResourceUnavailable) match by value.
- CertificateIdentifier no longer caches a certificate and the
CertificateValidator/CertificateValidation event is gone: the
configuration's CertificateManager owns trust lists and the loaded
application certificate and is disposed with the session; auto-accept
and the verify probe's capture-and-reject use AcceptError.
- new UserIdentity(X509Certificate2) is gone: X509 user identities use
X509IdentityTokenHandler with an on-demand PFX provider (still fails
fast on a missing file/key).
- DefaultSessionFactory takes the telemetry context; collections are
ArrayOf<T>; EventFilter select clauses and ContentFilter operands are
built immutably.
UserName identity (lmxopcua#880): SDK 2's default handler sends a
password over 64 bytes - every 65-character OtOpcUa API key - as an
RsaEncryptedSecret that SDK 1.5 servers reject with BadIdentityTokenInvalid
and SDK 2 servers accept only within a 5-minute clock window (#873). The
DCL now always uses LegacyRsaUserNameIdentityTokenHandler (ported from
OtOpcUa be28aa44), i.e. exactly what it sent on SDK 1.5, so it logs in to
both server generations during a mixed rollout.
Tests: the in-process secured test server is ported to SDK 2 and can
admit an API key; new tests cover the handler (mutation-checked: 5 of 7
fail with the SDK default handler), the value boundary, and a 65-char key
end to end over None/Sign/SignAndEncrypt with a wrong key reaching the
key check (BadIdentityTokenRejected).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A node frozen past failure detection + StableAfter, or both sides of a partition
that heals after each downed the other, used to stay a second active node until an
operator restarted one: the peer quarantines the removed UID and Akka.Cluster
1.5.62 never acts on ThisActorSystemQuarantinedEvent.
ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via
ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for
leader-auto-down only) watches two signals:
- S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer
is still a live member here, only this side can be shut out -> restart now.
- S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still
runs the incarnation this node removed. A healed partition yields only S2, on
both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then
tiebreak.
Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower
stays and restarts only if still split one grace later. Removed peers are probed
(Identify, 5s -> 60s back-off) since nothing else contacts them after mutual
removal. Restart = Cluster.Down(self) -> existing self-down path.
The Host now exits 75 after an unplanned ActorSystem termination (and sets it on
the Windows service before it reports stopped): SCM recovery with failureflag 1
only restarts a service that stops with a non-zero code, and the watchdog used
to exit 0. Shipped default strategy stays auto-down.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Auth 0.5.0 (all five packages) adds the passwordless ILdapGroupLookup. A new
singleton DirectoryGroupRefresher asks it for a session's current groups once
the new zb:lastgrouprefresh anchor (stamped at login) is older than
ScadaBridge:Security:LdapGroupRefresh:IntervalMinutes (default 15, Enabled
default true, CacheSeconds 60 per-node answer reuse). CookieSessionValidator
replaces the stored groups and re-maps roles; this reaches HTTP and live
circuits (once per interval, independent of the 10 s circuit refresh).
Failure policy: directory unavailable keeps the stored groups (warn, all paths
incl. fail-closed mutation checks); user not found / disabled / ambiguous /
no groups ends the session. Inert under BindMode=User (NotSupported),
LDAP disabled, or DisableLogin.
Auth 0.4.4 ApiKeyFailure.StoreUnavailable now answers 503 SERVICE_UNAVAILABLE
on the inbound API instead of a misleading 401.
Docs: Component-Security.md, components/Security.md, Component-Host.md,
register row 23 closed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Switch ScadaBridge from its own LoggerConfigurationFactory to AddZbSerilog
(ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.2.0) using the two 0.2.0 hooks:
- MinimumLevelGovernor: ScadaBridge:Logging:MinimumLevel, applied after
ReadFrom.Configuration so it outranks Serilog:MinimumLevel (never null, so the
floor never defers to the Serilog key; Override:* namespace entries still apply;
the one-shot conflict / unparseable-value warnings are unchanged).
- ConfigureLogger: the standby active-node exclusion filter (register row 71),
the configured node identity (the shared bootstrap stamps NodeHostname from
the machine name and skips empty values; an overwriting enricher keeps the
configured ScadaBridge:Node values verbatim), Enrich.FromLogContext and the
ExecutionLogCaptureSink.
ScadaBridgeLogging runs AddZbSerilog on a small pre-host builder and Program
installs the result as the static Log.Logger; both web hosts keep UseSerilog()
over it. One logger and one file-sink handle per process, usable before the
host is built (startup banner, warnings, pre-host Fatal), by SerilogAkkaLogger,
and after the host is disposed (shutdown line, unexpected-termination Fatal) —
AddZbSerilog's own DI registration would leave Log.Logger alone and have the
host dispose the logger.
LoggerConfigurationFactory is deleted (nothing else used it). Tests: existing
logging/standby/sink tests now run through the new pipeline, plus
ScadaBridgeLoggingParityTests (level precedence, Override:* still applies,
one-shot warning, configured identity incl. NodeHostname != MachineName and empty
SiteId, exact property set of a plain line, trace_id/span_id). Mutation-checked:
governor deferring, filter removed, identity enricher removed, FromLogContext
removed, capture sink removed each fail tests.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
AlarmConditionState.Severity mixed two conventions: OPC UA rows carried Part 9
Severity (higher = more severe) while MxGateway native alarms and computed
alarms carried a raw AVEVA-style priority (1 = most urgent). Every consumer
ranks higher as worse, so Worst Severity, the severity filter/sort and the
site event-log Error threshold picked the LEAST urgent MxGateway/computed
alarms.
- AlarmPriorityBands: the OtOpcUa GalaxyAlarmPriority table (lmxopcua#887,
b61f3e05), value for value: 1-250 -> 1000, 251-500 -> 750, 501-750 -> 500,
751-999 -> 250; <= 0 clamps to Critical, > 999 to Low.
- MxGatewayAlarmMapper bands the gateway severity (= AVEVA priority);
AlarmConditionStateFactory.ForComputed bands PriorityLevel. OPC UA unchanged.
- AlarmConditionState gains trailing Priority (int?): the raw source number,
exactly the pre-#129 Severity. AlarmStateChanged.Priority and the wire
AlarmStateUpdate.priority keep that meaning (no proto change); central derives
Severity by kind (SeverityFromPriority), so an older site decodes correctly.
- ScriptAlarm gains trailing Priority. The native MES 900-999 band and the MES
API Severity/MinSeverity/MaxSeverity key off Priority; rig seed scripts moved
from a.Severity to a.Priority, so MES output is unchanged.
- AlarmActor raise logs Error when the banded severity >= 700 (same threshold
as NativeAlarmActor): priority 1-500 Error, 501+ Warning (was raw >= 700).
- NativeAlarmActor upgrades pre-#129 native_alarm_state rows on rehydrate.
- Debug snapshot decode restores Condition.Priority from the row's priority.
- Docs: Commons, DCL, SiteRuntime, Communication, CentralUI, InboundAPI,
HighLevelReqs, MES alarm-status plan, CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb, .Replication and .Contracts at 0.3.7 (published
2026-11-12). Wire-compatible; no config or code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; its doc comment is updated. The
rig's site-a pair already carries one identical key on both nodes.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer resurrects deleted rows.
The survivor's ledger (sf_messages / site_events deletes) grows for the
length of the outage; optional TombstoneHardCap bounds it.
Deploy notes added to the topology + installation guides, the S&F
requirement and CLAUDE.md: stop/start the pair together, same ApiKey on
both nodes, ledger growth while a peer is down.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Sign/SignAndEncrypt sessions failed before activation with "ApplicationCertificate
for the security profile ... cannot be found" (BadConfigurationError): both
RealOpcUaClient.ConnectAsync and the verify probe built an empty CertificateIdentifier.
- New OpcUaApplicationIdentity builds the client ApplicationConfiguration for both
paths. Secured modes load the application certificate from a configurable own
store (subject + ApplicationUri), auto-creating a self-signed RSA-2048 cert only
when none exists. A persisted cert is reused across reconnect/restart and never
silently rotated: one that no longer validates (e.g. ApplicationUri mismatch)
fails closed. First-time creation is serialised per own store and logged at
Warning with the thumbprint. None mode is unchanged and never touches the store.
- New OpcUaGlobalOptions: ApplicationUri, ApplicationCertificateSubject,
OwnCertificateStorePath, AutoCreateApplicationCertificate,
ApplicationCertificateLifetimeMonths (validated).
- Owned resources: the private-key cert is disposed on disconnect, failed connect
and at probe end; sessions are now disposed after close, and a reconnect on the
same instance closes the previous session first.
- CertStoreActor wrote trusted server certs to the store ROOT, which the SDK never
reads (it uses {store}/certs), so an operator Trust never affected validation and
SDK-rejected certs never listed. It now uses certs/ and migrates legacy root files
on start.
- Rig: all site nodes (docker/, docker-env2/) put the four PKI stores on the
persistent /app/data volume with a pinned ApplicationUri.
- Tests: in-process strict-trust StandardServer (test-only Opc.Ua.Server 1.5.378.106)
exercising the real adapter through ConnectAsync and VerifyEndpointAsync for Sign
and SignAndEncrypt: allow + read + reconnect + restart on one identity, deny when
the client doesn't trust the server (BadCertificateUntrusted), deny when the server
doesn't trust the client (BadSecurityChecksFailed), None untouched, concurrent
provisioning creates one cert, URI mismatch fails without rotating, auto-create off.
Against the pre-fix client these tests fail with the exact #121 error.
#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.
#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
registration is declared inside the launcher's and executor's try, so it ends before the catch
blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.
Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.
Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LogContext is an AsyncLocal and does not cross an Akka message hop, so the run's id now rides the
message: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) on
SetStaticAttributeCommand, AttributeValueChanged (not for an inferred writer), ScriptCallRequest and
the four RouteTo*Requests, plus a new additive SourceExecutionId on GetAttributeRequest,
WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest and WriteTagBatchRequest.
ScriptRuntimeContext stamps its id on the requests it sends, the Instance Actor hands it on to the
DCL, and the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in
AroundReceive (ExecutionLogContext.PushFor: one type test when the message names no execution). The
inbound executor pushes the request's ExecutionId for the method execution.
Tests (TestKit): Instance Actor line carries the message's id, no leak from the sender's ambient
context, scope ends with the message, per-write ids inside a batched SetAttributes, ids forwarded on
DCL writes, ScriptRuntimeContext stamps; DCL adapter lines on single/batch writes carry the id;
inbound executor lines carry the request id; Commons mapping + no serialized member. Each mutation-
proved. Still open on #56: a queryable per-execution sink and a Flow Debugger section. The DCL
actors' own Akka _log lines bypass Serilog (filed #120).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Bundles did not record that a template was a derived slot template, so the
importer re-created every slot as a BASE template named after the slot.
- Export: TemplateDto gains an additive, null-omitted `Derivation`
(owner path + base path by stable names); a derived template carries only
its own rows (overrides/additions, not inherited placeholders). The closure
pulls in each derived template's base/owner; base orders first.
schemaVersion 1.1 -> 1.2, bundleFormatVersion stays 1.
- Import: base templates go through the ordinary passes; derived templates are
ordered topologically (DerivedTemplatePlan; cycle / duplicate / malformed =
preview Blocker + SemanticValidationException before anything is staged),
their owner and base resolved by path (#100 root rule), and the slot is
created with TemplateService.AddCompositionAsync, synced in place when it
already exists with the same base, or deleted+recomposed when the base
differs; own rows applied as overrides, then ResyncInheritedMembersAsync.
Missing owner/base fails that record (BundleImportDerivedTemplateUnresolved
audit + ImportResult warning), never creating a base in its place.
- Preview/diff: derived rows are keyed by qualified name and compared against
the target slot at the same owner path (own rows + base), so an unchanged
slot is Identical and a same-named base template is never matched.
- Pre-#118 bundles import exactly as before (new fixture
pre-118-derived-plaintext.scadabundle, written by a13aabc7). The field is in
content, pinned by the signed manifest's contentHash.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every downing decision of LeaderAutoDowningProvider is now logged under a
[leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable),
unreachable observed + its window (Warning), skipped because already Down/Exiting,
pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept
pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own
AutoDown logged only the final Down. Decision semantics are unchanged.
New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests),
built from production BuildHocon with self-first seeds:
- both nodes cold-start together -> one cluster, nobody downed past FD + window;
- lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up,
then the peer cold-starts and joins;
- flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the
window, repeated past it in total -> never downed; positive control asserts the cuts
really produced unreachability;
- partition held past the window -> each side downs the other, both stay Up and
leader (the accepted dual-active trade);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider,
only leader-auto-down loads the in-repo provider.
Unit: flapping never downs; a flapping member that finally stays unreachable is downed
exactly once a full window after its LAST unreachable; each decision log line.
Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable
ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping
test fail; DOWNING log dropped -> log test fails; provider never downs -> partition
test fails; auto-down miswired to the new provider -> switch-off theory fails.
Docker failover drill not run (it would rebuild the shared scadabridge:latest image
and reuse the shared rig volumes while another agent works in the repo); filed as
deploy gate ScadaBridge#116. Default remains auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TemplateLibrary.GetIdByNameAsync was first-row-wins, so a derived slot template
could shadow a same-named base. Now: exact name, base first, derived-only name
only when unique, otherwise unresolved (composition audit says ambiguous).
Tracked-template indexes are base-first too.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
OtOpcUa's out-of-process script runner publishes 0x80040000 with a null value.
- OpcUaDataConnection: logs entry/recovery once per tag, names the symbolic
status in read errors; never a disconnect (not in the reconnect status set).
- InstanceActor: a Bad update with no value is a quality-only transition
(keep value, quality Bad, stream publish, no script/alarm/waiter re-eval),
matching the connection-drop rule. Bad with a value is unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
#113: AuditLogPartitionMaintenance (monthly SPLIT RANGE) and the
backfill-source-node endpoint now use the AuditLog:MaintenanceConnectionString
login when set (unset = unchanged). Migration AuditMaintenanceGrants gives
scadabridge_audit_purger ALTER ANY DATASPACE and column-level UPDATE (SourceNode).
#114: archive cleanup deletes the writer's own *.tmp files and manifest-less
data files older than 24 h inside {Directory}/{YYYY} (DeletedOrphan, dry-run aware).
#115: head-export failures bump scadabridge.audit.chain_head_export.failures and
Degrade the /health/ready entry audit-chain-head-export (failing or stale > 2 intervals).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions recorded: the archive destination stays a directory/UNC
(no object-storage client; share + permissions named at enablement),
parquet-tools-only reading (no re-import), unsigned like the chain (#107).
AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed
AuditLog:RetentionDays, counted from the archived month's end) and
DeleteSupersededStaleArchives (default true) run after every purge tick.
A retried purge's stale Archived archive is deleted only when a later,
verifying, Purged archive of the same partition provably holds every row:
same prior anchor and the stale chain-end hash at its chain-end sequence
(row or tombstone), every unchained row present by EventId; otherwise it is
reported KeptStale. A sole copy left at Archived is never touched. Deletion
removes only the archive name's own three files in its folder, manifest
first, never a manifest-supplied path. `audit archive cleanup [--dry-run]`
= CleanupAuditArchives (Administrator).
AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy,
sole copy, retention and the path guard; each safety rule mutation-checked.
Follow-ups filed: #113, #114, #115.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 1 (owner decision: no signing). AuditLog:HashChain:HeadExport (default
off): both centrals periodically read every month's chain head (LastSeq,
LastHash, purge anchor, chained rows, tombstones; one snapshot under the
chain lock SHARED) and append the changed heads to a sink the DB login cannot
write - File (append-only JSONL per node per day, local or UNC) or Syslog
(RFC 5424 over UDP / TCP octet counting). `audit verify-heads --path` reads
those records back (file or syslog-collector lines) and the new
VerifyAuditChainHeads command checks each against the database: Matches /
Extended / Purged, or Mismatch / Regressed / MissingHead / MissingLink /
CountMismatch (exit 3). AuditChainHeadStoreTests forges a chain with the
production hasher: verify-chain says Intact, verify-heads says Mismatch.
Item 3. AuditLog:MaintenanceConnectionString (${secret:}-capable, unset =
today's single login) runs the chain job, chain verify / head export, the
partition switch-out + archive and the per-channel purge on
AuditMaintenanceContext (keyed audit-maintenance repository/archive, purge
actor prefers them). AuditMaintenanceLoginTests runs the hardened setup on
SQL Server: writer (datareader/datawriter/audit_writer) ingests and is
refused chaining and deleting; the chainer+purger login does the full cycle.
Two-login grants documented in Component-AuditLog.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the
twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId)
on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred
(EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve
indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production
scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql.
The partition-switch purge staging table now reads the live clustered key
order from sys.index_columns, so it switches either shape (old binaries
hard-code the old order: ship with the binaries, both centrals stopped).
Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward
clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14;
FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the
staging key (mutation: hard-coded old key fails SWITCH) and the read ratio.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions on #110: one signing key per environment; TrustedKeys stay hand-edited;
Signing:ProductionBoundary (new, default off) logs a startup warning and reports a Degraded
bundle-signing-policy /health/ready entry while RequireSignedBundles is off; the private key
is read via ${secret:} (proved end to end, never logged); cluster pull decided not to be
built; differential deletions stay deferred. Adds docs/operations/bundle-signing-runbook.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. PendingUpdatesEvaluator re-flattened every
deployed instance on each cache-TTL expiry even when nothing had changed.
- Migration AddConfigGraphRowVersions: a non-concurrency rowversion column
+ index on the 18 configuration tables the evaluation reads (Templates +
5 member tables, Instances + 4 override/binding tables, DataConnections,
DeployedConfigSnapshots, Sites, SharedScripts, ExternalSystemDefinitions,
ExternalSystemMethods, DatabaseConnectionDefinitions). SQL Server only in
the model; SQLite test contexts are untouched. Script:
docs/plans/sql/AddConfigGraphRowVersions.sql. Joins the coordinated
central-pair migration window (#107 item 4).
- IConfigGraphWatermarkReader / ConfigGraphWatermarkReader: ONE query of
MAX(RowVersion) + COUNT_BIG(*) per table (count catches deletes, which
raise no rowversion). rowversion is engine-bumped for any writer, so a
peer central node's edit moves it — not the per-process watermark MG-3
removed.
- The secret store (outside the DB) gets its own stamp: SHA-256 over
ListAsync(includeDeleted: true) metadata.
- Evaluator re-serves the last CLEAN snapshot while watermark + secret
stamp are unchanged; every PendingUpdatesFullEvaluationEvery'th (default
10, validated >= 1, 1 = off) is full regardless — the safety net for a
same-KEK re-wrap or an input table missing from the set. Always full when
the token is unavailable/unreadable, after an evaluation that could not
compute a site's artifact hash, on Re-check, and after a deploy completion.
It only re-serves a real earlier answer, never manufactures one.
Tests: SQL Server (real migrated DB) — token stable without writes, moves
for a peer-connection INSERT/UPDATE and a non-max DELETE, moves for a local
EF write, a detached Update still saves (not a concurrency token), and a
guard that every table the evaluator's repository reads touch is covered.
Evaluator — reuse without flatten (re-served answer still shows the stale
instance), watermark move, 4 secret changes, every-Nth, 5 no-token modes,
unclean evaluation not reused, cache Re-check and deploy completion force
full. Mutation-proven: token compare, clean check, cache force flag, row
count, and an uncovered read each turn tests red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. An inbound API method has no site
affinity, so the F2 route-time scope only bounded routed SITE traffic;
the method's central-side effects (Database.*, ExternalSystem,
notifications) still ran for a scoped Deployer. FlowDebugRunner now
refuses FlowDebugRunMethodCommand for a site-scoped caller (site ids,
not Administrator) before the script executes, audits the refusal as a
failed InboundRequest under the operator's name, and raises
SiteScopeViolationException -> ManagementUnauthorized, so the Central
UI and the CLI (same /management command) share the gate.
The page disables the "Inbound API method" entry for scoped Deployers
with the reason (tooltip + inline note), starts them on script entry,
and ignores ?methodId= / a stored method choice. Replay and the
instance-scoped script / value-change Live runs are unchanged. The
route-time scope is kept as defence in depth.
Tests: scoped refused (in- and out-of-scope routing; script never runs,
refusal audited), unscoped/Administrator allowed, scoped Replay allowed,
scoped in-scope value-change Live still runs; bUnit for the disabled
entry, the drill-in, and the unscoped control. Mutation-proven (gate
and UI flag reverted -> 4 red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Exports can now carry a detached ECDSA P-256 signature over the exact
manifest.json bytes in a new optional manifest.sig zip entry. The manifest
already pins the content hash, so a verified signature covers the whole
bundle, including the plaintext source environment the confirm step relies
on. Off by default (ScadaBridge:Transport:Signing): with the defaults,
exports are unsigned and imports accept signed and unsigned bundles.
- The bundle format does not change: bundleFormatVersion stays 1,
manifest.json and the AES-GCM associated data are untouched, and older
importers open entries by name and ignore manifest.sig. Three entries fit
within the default MaxBundleEntryCount of 4.
- Import policy runs after the content-hash check and before any passphrase
work. A signature that fails to verify against a trusted key is always
refused. RequireSignedBundles additionally refuses unsigned bundles and
bundles signed by an untrusted key. A refusal is BundleSignatureException
(Commons), shown on the UI upload step and on CLI preview/import.
- The signature status, key id and signer are recorded on BundleSession and
shown in the import wizard. They are also returned on PreviewBundleResult
and recorded on the BundleImported audit row. BundleExported records
Signed/SigningKeyId.
- Key material is pluggable. IBundleSigningKeySource and IBundleTrustStore
are TryAdd'd; the defaults read Signing:PrivateKey (meant to be a
${secret:} reference) and Signing:TrustedKeys. Key ids are SPKI
fingerprints. Startup validation rejects unparseable or non-P-256 keys.
- Compatibility proof: the checked-in fixtures pre-signing-plaintext and
pre-signing-encrypted were exported by origin/main 3ff463b6, before this
change, and are imported end to end with signing off and with signing
configured.
The design leaves key-per-env vs per-org, trust distribution and rotation,
and the production default for RequireSignedBundles open. These are listed
in Component-Transport.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Per-hop current state with code evidence (browser/Traefik/Kestrel, CLI,
central<->site gRPC + PSK, LocalDb sync, Akka remoting, OPC UA /
MxGateway), the recommended mode per hop (Kestrel HTTPS + ALPN, gRPC over
TLS with validation and the PSK retained, Akka DotNetty SSL with a cert
per node and mutual auth), certificate issuance/storage/rotation, rig and
wonder impact, a staged rollout, rollback, a test plan and 13 open owner
decisions. Findings: the central ASPNETCORE_URLS re-bind drops the
scheme (an https URL binds plaintext); Akka.NET has no secure-cookie, so
mTLS is the only remoting authentication. Component-ClusterInfrastructure
documents the new RemotingTls option.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET
1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the
leader among the reachable members downs a peer unreachable for StableAfter; a
non-leader records it as pending and downs it on becoming leader; reachable-again
or removed members are forgotten; Down/Exiting members are skipped. Its window is
carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no
deprecated auto-down-unreachable-after key is emitted and Akka's boot-time
"keep-majority will be used instead" warning goes away. DownRemovalMargin reads
akka.cluster.down-removal-margin exactly as AutoDowning does (default off).
New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon
and ClusterOptionsValidator. The default stays "auto-down"; auto-down and
keep-oldest HOCON is byte-identical to before. Placed in Host, not
ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays
Akka-free).
Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec
cases (leader-only, pending-on-leader-change, losing leadership, reachable again,
removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests
clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node
cluster plus a test that Akka actually loaded the provider and its downing actor.
Mutation-checked: a provider that never calls Cluster.Down fails both failover
tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed
mutations each fail the unit tests.
Remaining gate before switching the default: the docker failover drill
(DRILL_MODE=active and standby mode) on leader-auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Component-AuditLog Security & Tamper-Evidence describes the per-month chain,
the chaining job and its lock, both purge paths (checkpoint anchor,
tombstones), pre-existing rows, verification and what an unsigned chain does
not prove. alog.md §11.4/§16, CLAUDE.md, HighLevelReqs AL-11/12,
Component-CLI, Component-ManagementService, docs/components/CLI.md and the
CLI README drop the "no-op placeholder" wording.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
(a) Inbound API compile state (#75) is per central node: a recompile cleared the
known-bad record only on the node that answered, and the peer kept refusing the
unchanged script with SCRIPT_COMPILE_FAILED (the per-request revision check only
notices a changed script). Create / update / delete (ManagementActor post-commit),
RecompileApiMethod, and API methods written by a Transport bundle import now publish
InboundApiMethodInvalidated on the central cache-invalidation topic (#100 item 3).
The new InboundApiCacheInvalidationHandler drops that method's compiled handler,
known-bad record and cached ApiMethod row on the peer, which compiles the current
text on its next request.
(b) docs/components/InboundAPI.md no longer claims a direct DB edit stays stale
until restart: the per-request revision check picks it up once the node's cached
row expires (ApiMethodCacheTtl, 5 s). Component-InboundAPI, Component-Host and the
#85 cache audit describe the peer invalidation.
(c) Decision: the import's locked-override warnings keep EXACT (ordinal) name
matching, because FlatteningService matches override names ordinally. An override
differing from a locked member only by case never reaches that member's lock check
(the flattener drops it as unresolved), so a "targets a LOCKED member" warning for it
would be false. The comparer is now named (LockedOverrideNameComparer) and documented,
and pinned by tests on both sides (flattener + import).
Tests: the peer's known-bad record is cleared by a recompile (and validate leaves it
alone); create/update/delete and the import publish the invalidation; the handler
drops only the named method; composition root registers the handler; case-only
override gets no lock warning; the flattener matches override names exactly.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast.
It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation,
carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated,
RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated)
in an envelope stamped with the publisher's process-unique origin id.
- Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until
the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which
skips its own node's envelopes and hands the peer's to every registered
ICentralCacheInvalidationHandler (a failing handler does not stop the rest).
Replaces SitePskInvalidationSubscriber.
- Migration of #96: SitePskRotationInvalidator publishes through the bus and is the
PSK handler; SitePskInvalidated keeps its type name and is also published bare on
site-psk-invalidation, which the subscriber still listens to, so a mixed-version
central pair keeps exchanging PSK invalidations. Remove after one release.
- Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping
invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on
the peer instead of after RoleMappingCacheSeconds.
- Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9
ManagementActor handlers and the Transport bundle importer publish
ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.)
TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and
subscriber unit tests (origin filter, legacy topic), peer-node tests through the real
caches in ManagementService.Tests, importer publish, composition root, and a real
two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests)
proving every message type crosses with the production serializer. Docs: Host,
Communication, Security, ManagementService, ESG, and the #85 cache audit.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The bulk SetInstanceNativeAlarmSourceOverridesCommand handler always wrote the
entry's IsMesReportableOverride, so a CLI import of a legacy 4-column CSV (no
MesReportable column) reset every imported source's MES override to inherit. The
Central UI avoided it only by looking up the stored flag client-side.
NativeAlarmSourceOverrideEntry gains an additive trailing KeepStoredMesReportable
flag: when set, the server leaves an existing row's MES override untouched (a new
row inherits) and ignores IsMesReportableOverride; unset keeps the old meaning
(null = clear to inherit), so older senders are unaffected. The CLI and the Central
UI now build their entries through one rule (ToOverrideEntry on each row type):
no MesReportable column => KeepStoredMesReportable. CLI README and
Component-CentralUI updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The data plane answers a broken inbound method with the generic "Script
compilation failed for this method"; the Roslyn diagnostic reached only the
central log, and the known-bad cache could not be observed or reset per method.
- InboundScriptExecutor keeps the diagnostics with each known-bad record and
exposes GetCompileState(name, currentScript): IsCompiled / IsKnownBad /
LastCompileError, judged against the current script text (a record for older
text is reported as not compiled yet).
- Management: ValidateApiMethodCommand (compile the stored script, return
ApiMethodCompileResult, register nothing, cache untouched) and
RecompileApiMethodCommand (CompileAndRegister without re-sending the entity,
drops the node's ApiMethodCache row, audited "Recompile"). Both Designer, like
UpdateApiMethod; dispatched through the shared ApiMethodCompileOperations.
GetApiMethod keeps the entity fields top-level and adds isCompiled,
isKnownBad, lastCompileError and compileStateNode. Compile state is per
central node and every result names the node.
- CLI: api-method validate --id, api-method recompile --id.
- Central UI: the API-method edit page shows the saved script's compile state
and last error, with Validate and Recompile actions (same operations, same
Designer gate, fresh DI scope).
- RequiredRoleMatrixTests entries; ManagementSurfaceParityTests covers the new
CLI leaves; ApiMethodCompileDiagnosticsTests + an executor CompileState test.
- Docs: CLI README, Component-CLI/ManagementService/InboundAPI, components/*.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
New AuditChannel.Security with AuditKind.CertTrust / CertTrustRemove (appended, ordinals stable).
Both central entry points - the management actor handlers and the Central UI CertManagementService -
write a best-effort ICentralAuditWriter row after the site outcome is known (Delivered when all
reachable site nodes acked, Failed otherwise, incl. relay timeouts), naming actor, site, thumbprint
and connection. Shared builder: CertTrustAuditEvents.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:
- the secret is written through /admin/secrets: central replaces the secrets
library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
publishes SitePskInvalidated on a new DistributedPubSub topic
(site-psk-invalidation) so the peer central node drops its copy and its
resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
WithSiteCredentials wraps every per-site channel's handler in
PskRejectionRefreshHandler, so the next call re-reads the key (covers
out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.
Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Replace the private SitePairActiveNodeHealthCheck (503 whenever no ActorSystem, no data)
with the shared ZB.MOM.WW.Health ActiveNodeHealthCheck constructed with the site-{SiteId}
role: Degraded (200) during startup per the Health spec, and it publishes
selfAddress/activeRole/activeNode so the Overview dashboard can name the active site node.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audit (docs/plans/2026-11-09-central-cache-staleness-audit.md) of every central in-process
cache holding configuration or secret state, with its staleness bound on the peer central
node. Every config cache with peer staleness is TTL-bounded (5-60 s) and Traefik routes to the
active node only, so no peer case was clearly a bug and no cross-node broadcast is added; the
doc names the existing DistributedPubSub mediator (health replication topic) as the mechanism
if one is wanted. Owner decisions listed: RoleMappingCache 30 s revocation lag on the peer,
SitePskProvider caching a PSK forever across an in-place rotation (both nodes), and the #83
ExternalDefinitionLookupCache 10 s bound.
Fixed: ManagementActor API-method create/update/delete invalidated only the compiled handler,
not ApiMethodCache, so the node that made the change kept serving the pre-change row for up to
the 5 s TTL (a deleted method stayed callable; a cached not-found hid a new one). The handlers
now drop the entry post-commit, like the role-mapping cache.
Tests: ApiMethodCacheInvalidationTests (create/update/delete fail without the fix; a
negative control checks other methods stay cached).
(ScadaBridge#85)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The InboundAPI carried its own IActiveNodeGate (implemented by Host/Health/ActiveNodeGate)
whose rule already matched the shared Health 0.3.0 AkkaActiveNodeGate — both route to
ClusterActiveNode (oldest Up member). The private interface and implementation are deleted;
InboundAPI references ZB.MOM.WW.Health (0.3.0, already pinned) and InboundApiEndpointFilter,
AuditWriteMiddleware, InboundMethodWarmupService and FlowDebugRunner consume
ZB.MOM.WW.Health.IActiveNodeGate. The Central branch of Program.cs registers
AkkaActiveNodeGate (unscoped, like the active-node health check).
Warm-up behaviour is kept: the shared gate answers false while no ActorSystem resolves, while
this member is not Up, and on any cluster-init fault. It resolves the ActorSystem through the
existing DI singleton bridge (GetOrCreateActorSystem), the same path the health probes use.
Tests: CompositionRoot asserts the shared gate type (fails with the registration removed);
the evaluator wiring test now drives AkkaActiveNodeGate; new warm-up test (no ActorSystem ->
not active). Nothing was missing from the Health 0.3.0 package.
(ScadaBridge#80)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
`health pending-updates` (#63) called IPendingUpdatesEvaluator.EvaluateAsync directly
from ManagementActor because the cache lived in CentralUI, so every CLI call was a full
uncached fleet flatten. IPendingUpdatesCache/PendingUpdatesCache now live in
DeploymentManager.PendingUpdates and are registered by AddDeploymentManager; the
SingleFlightMemo<T> primitive moves to Commons.Caching (public) so both CentralUI's KPI /
alarm-summary caches and DeploymentManager can use it. HandleGetPendingUpdates reads the
shared cache; the UI circuits and the CLI now share one memo per node.
Tests: ManagementActor repeated-calls test (fails against the direct evaluator call:
2 evaluations instead of 1), AddDeploymentManager singleton-registration test; the cache
and memo tests move with their types.
(ScadaBridge#91)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- ApplyTemplatesAsync/PreviewAsync template lookups use a case-insensitive,
never-throwing comparer (SQL Server collation); base template wins over derived.
Base names are already unique case-insensitively via the filtered unique index
on Template.Name, derived templates are not covered by it.
- Preview reads database connections, notification lists and API methods with one
GetAll query each instead of one by-name query per bundled row.
- Component-TemplateEngine.md: the flattener does not call GetAllTemplatesAsync;
it loads chains via GetTemplateWithChildrenAsync.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Measured first: a scratch scaling probe (seed -> export -> load/preview/apply
into a fresh target and over itself, in-memory provider, every repository
call counted, DetectChanges counted via ChangeTracker.DetectingAllChanges)
showed load, preview and the instance pass scale linearly (500 / 1000 / 2000
instances apply in 0.13 / 0.27 / 0.35 s), but the post-flush template
rewire passes were quadratic: 100 / 400 / 800 composing, derived templates
applied in 0.5 / 8.1 / 26.5 s, with DetectChanges running 5N+10 times and
GetTemplateWithChildrenAsync 2N+2 times.
Cause: ResolveAlarmScriptLinksAsync, ResolveCompositionEdgesAsync and
ResolveInheritanceEdgesAsync found each template with
_dbContext.Templates.Local.FirstOrDefault(...) - per template, and per
composition edge / base lookup. Every DbSet.Local access runs a full
DetectChanges over everything the import has staged. The first two passes
also re-read every template's children one query at a time (a split query
per call on SQL Server).
Fix (no bundle-format or semantic change):
- each pass indexes the tracked templates by name once
(SnapshotLocalTemplatesByName; ordinal, first wins like FirstOrDefault).
These passes never stage a template or rename one, so the index stays
exact for the pass;
- the alarm and composition passes read their templates' children in one
batched GetTemplatesWithChildrenAsync, matched back by the stub's id;
- the alarm pass only loads templates whose bundled alarms name an
on-trigger script (the only ones it can rewire).
Result on the same probe: 100 / 400 / 800 templates apply in
0.13 / 0.62 / 0.64 s; DetectChanges is a constant 9; per-template children
reads 0, batched reads 1 (add) / 2 (overwrite).
Evidence: ImportApplyScalingTests (Transport.IntegrationTests) exports 5 and
40 composing, derived templates with alarm -> script links, imports each
into an empty target (Add) and again (Overwrite), and asserts the imported
inheritance / composition / alarm-script links, zero per-template children
reads, at most 3 batched reads, and an identical DetectChanges count for 5
and 40 templates. Fails on the old importer (12 per-template reads), and
with a per-DTO Local snapshot reintroduced in one pass (DetectChanges 51
vs 16). Transport unit (198) and integration (138) suites pass.
Not changed: preview's per-name lookups for database connections,
notification lists and API methods stay per-name - moving them to an
in-memory ordinal match could change name matching under a case-insensitive
SQL collation.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DeploymentService.TryPurgeTerminalDeploymentRecordsAsync rate-limited its
opportunistic retention sweep with a process-wide static tick field, so only
one test in the assembly could drive the sweep and its swallow-on-failure path
was untested.
The limit now lives in TerminalPurgeRateLimiter: a TimeProvider-driven
compare-and-swap claim (MinInterval 6 h) registered by AddDeploymentManager as
a DI singleton, so every scoped DeploymentService still shares one limit per
node. DeploymentService takes it as an optional trailing constructor parameter;
a hand-built instance gets its own limiter. The sweep's cutoff now comes from
the same clock. Behaviour is otherwise unchanged.
Evidence (TerminalPurgeRateLimiterTests):
- claim once per interval on an injected clock; 16 concurrent claimants ->
exactly one wins;
- two services with their own limiters both sweep, two sharing one sweep once;
- cutoff = injected now - retention;
- a failing artifact batch is logged as a warning with the exception and
swallowed, the already-purged terminal rows are still reported, the next
call inside the interval does not retry and the first call after it does;
a first-batch failure returns 0 and skips the artifact purge;
- AddDeploymentManager registers the limiter as a singleton shared across scopes.
Mutation-checked: a process-wide static limiter fails 5 tests (incl. the
existing artifact-purge test); narrowing the catch fails both failure tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The last two whole-library template reads are gone. Both fed the
inheritance-only TemplateInheritanceResolver, which walks one template's
parent chain and never follows compositions.
- TemplateService.ResyncInheritedMembersAsync (operator repair, CLI + editor)
now reads #86's GetTemplateReconcileClosureAsync (derived subtree + the
template's chain, tracked). When nothing derives from the template the
closure is empty, so it falls back to the new tracked
GetTemplateChainAsync - resyncing a leaf reads only its own chain.
- ManagementActor.HandleGetResolvedTemplateMembers now reads
GetTemplateChainAsync(id, trackChanges: false): the chain alone, untracked,
with script bodies (the resolver projects script code, so #66's body-free
analysis closure would not be equivalent).
The chain walk is factored out of TemplateReconcileClosure into
CollectChainAsync (one load per level, stops on parent cycles and dangling
parents) and shared by both.
Evidence:
- TemplateChainReadTests (#86 fixture): for every template plus an unknown
id, the closure-backed resync issues exactly the writes, audit row and
result of the whole-graph resync with GetAllTemplatesAsync wired to throw;
the resolver answers identically from the chain and the whole graph.
17 of these fail against the old service; 13 fail with the leaf chain
fallback removed (mutation-checked).
- TemplateChainRepositoryTests (SQLite, recorded SQL): a 4-deep chain loads
4 templates / 8 attributes instead of 46 / 92, one split-query set per
level, untracked on request and tracked for the resync.
- ResolvedTemplateMembersReadTests: the actor answers from the untracked
chain read and never calls GetAllTemplatesAsync (fails on the old actor).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Health page gains PendingUpdatesHealthTiles (stale instances / sites with stale
artifacts, Deployer+Administrator only, site-scope narrowed, shared cache). New
GetPendingUpdates management command (Deployer) and 'health pending-updates' CLI
leaf. Flatten/cache computation untouched; the command reads
IPendingUpdatesEvaluator directly because the UI cache is not reachable from
ManagementService.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
New SiteWriteFailureBadges component renders warning badges for
SiteEventLogWriteFailures, SiteAuditWriteFailures and AuditRedactionFailure
when > 0 on online sites. The CLI 'health summary' already serialises the full
latest report, so these counters are visible there unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TemplateService.ReconcileDescendantsAsync - run after every base member
add/update/delete, template and native-alarm-source alike - loaded the whole
tracked template graph (every template, member and script body) on each call,
which after #66 was the dominant cost of those paths. It now calls the new
ITemplateEngineRepository.GetTemplateReconcileClosureAsync: the same tracked,
fully-included shape as GetAllTemplatesAsync, restricted to the edited
template's derived subtree plus its own inheritance chain. The reconcile is
inheritance-only (TemplateInheritanceResolver never follows compositions) and
every descendant's chain runs through the edited template, so that set is
everything it reads.
The walk (Commons TemplateReconcileClosure, the descendant-direction twin of
#66's TemplateAnalysisClosure) loads one children batch per subtree depth and
one batch per ancestor level, terminates on parent cycles and stops at
dangling parents. A template with no children - the common case - costs one
empty probe and loads nothing.
Evidence:
- TemplateReconcileClosureTests: for every template of a graph with a deep
subtree, an ancestor above the edited template, a composed module,
look-alike unrelated trees, a parent cycle and a dangling parent, the
closure-backed reconcile issues exactly the writes (add/update/delete,
audit, save) of the old whole-graph reconcile, and the resolver answers
identically for every descendant. Fails with the old service (whole-graph
load wired to throw) and with the ancestor walk removed (mutation-checked).
- TemplateReconcileClosureRepositoryTests (SQLite, recorded SQL): 5 templates
/ 10 attributes instead of 47 / 94, tracked with script bodies; a leaf costs
one probe query and tracks nothing.
- TemplateServiceTests: attribute add + update propagate two levels down with
GetAllTemplatesAsync wired to throw; existing propagation tests now run
against the closure via ServeReconcileClosureFromFullGraph.
ResyncInheritedMembersAsync (operator action) still reads the whole graph.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SiteAuditBacklogSnapshot gains an optional trailing RejectedCount (gauge of
Rejected forward-state rows, read from the site store), carried on the wire as
SiteAuditBacklogSnapshotDto.rejected_count = 4 (additive; old sites read as 0,
old central ignores it). Health page shows an 'Audit rows rejected' badge.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Each cached client records when it was last handed out; a periodic sweep (half of the new
CommunicationOptions.GrpcClientIdleTtl, default 30 min, 0 disables) disposes and removes entries
unused for the TTL that have no live subscription (SiteStreamGrpcClient.ActiveSubscriptionCount).
Lookup and the sweep's check-and-remove share one lock so a handed-out client is never disposed
under its caller; an evicted client is re-created on next GetOrCreate.
Joseph Doherty