Removes the #96 compatibility path that the unified central-cache-invalidation
topic (#100 item 3, 153609ef; compat 3a31dd2d) carried for one release:
- CentralCacheInvalidationBus no longer publishes a bare SitePskInvalidated
on site-psk-invalidation; LegacySitePskTopic constant deleted.
- CentralCacheInvalidationSubscriber no longer subscribes to that topic or
handles a bare SitePskInvalidated; it applies envelopes only.
central-cache-invalidation behaviour is unchanged (one envelope per
invalidation, origin filtering, per-handler fault isolation).
Gate met: 8e32f353 (carries 153609ef and 3a31dd2d) runs on every central
node - DEV vd03 (record 581e25ca) and QA vq03/vq04 (record 8ba5d5d7).
Mixed-version / rolling-upgrade constraint: a central node built before
153609ef listens only on site-psk-invalidation and no longer receives PSK
invalidations from a newer peer (only its TTL and the site-rejection refresh
bound the staleness). Mixed pairs across that boundary are no longer
supported; central pairs are stopped and started together anyway.
Documented in Component-Host.md and Component-Communication.md.
Tests: the legacy publish/subscribe tests are replaced by
PskRotation_OnOneNode_ReachesThePeersCaches_ThroughTheUnifiedTopic (node A
publishes, node B's PSK cache and secret-resolver cache are invalidated, and
nothing else is published) and a bare-message-ignored test with an envelope
positive control. The two-node cluster test now asserts the PSK invalidation
arrives exactly once and the publisher applies nothing.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor (#140) stays.
GracefulStopWithOpenSyncStreamTests' positive control passed the stop
in ~900 ms under 0.3.8, because the library alone now ends the stream.
The control registers LocalDbSyncService without the host lifetime (the
0.3.7 behaviour), so it still reproduces the 8 s drain; verified it
fails on 0.3.8 without that change.
Release build 0 warnings. Host.Tests 733/733, IntegrationTests 133/133
(incl. the LocalDb site-pair convergence tests), Communication.Tests
843/843, HealthMonitoring.Tests 131/131, SiteEventLogging.Tests 118/118.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A graceful stop of a site node whose peer holds an open LocalDb sync stream
left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906,
scadaproj#57). WebApplication stops the web server before AkkaHostedService,
whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited
the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7
observes only the call token, which Kestrel cancels at the END of the drain.
CancelAllStreams only covered SiteStreamGrpcServer's own streams.
- AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs
before any hosted service's StopAsync (Kestrel's included): marks the stop
intentional, raises ApplicationStopping (so CancelAllStreams still precedes
actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason),
bounded by the host shutdown token. StopAsync returns the same completed
shutdown. Exit codes unchanged: planned stop exits 0; only a termination
outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting
here, so no ClusterLeavingReason/-1 exit-code path is involved.
- ShutdownStreamCancellationInterceptor, innermost on both the central and the
site AddGrpc pipelines: every streaming call's token is linked with
ApplicationStopping; new streams are refused Unavailable while stopping.
Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned.
Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real
WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor
+ a real LocalDb SyncBackgroundService peer + the production AkkaHostedService
forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp
the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms;
after 960 ms stop, member removed at 950 ms, drain began at 959 ms.
Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor
passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync
does not raise ApplicationStopping; lifecycle interface dropped; site pipeline
loses the interceptor.
Release build of the solution: 0 warnings. Host.Tests 733/733,
IntegrationTests 133/133, Communication.Tests 843/843.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Owner decision on #107 (2026-11-13): enabling AuditLog:HashChain must not
chain the historical backlog. main chained every unchained row oldest-first.
- The first chaining pass on a pristine database (no head, no chained row)
writes a chain-start anchor: AuditLogChainTombstone (cutoff month,
ChainSeq 0), PurgedAtUtc = DB SYSUTCDATETIME(), RowHash =
SHA-256("ScadaBridge.AuditLog.ChainCutoff.v1" || YYYY-MM || ticks).
Written once under the exclusive chain lock, so both centrals agree
whatever their clocks/config. No migration: existing table + grants.
- Only rows with OccurredAtUtc >= cutoff are chained. The anchor hash is
the cutoff month's genesis, binding the cutoff into the chain.
- AuditLog:HashChain:ChainBacklog (default false) restores chain-everything,
read only on the very first enablement. Legacy chains (heads, no anchor)
keep chaining everything.
- verify-chain splits unchained rows into UnchainedRows vs PreCutoffRows
(+ ChainFromUtc); new breaks ChainedBeforeCutoff (row re-dated before the
cutoff, or cutoff moved forward) and CutoffAnchorInvalid (anchor hash /
month / multiplicity). Archive verify uses the anchor from its own
tombstone file or the DB.
- Partition switch-out keeps the anchor (deletes ChainSeq > 0 only); the
head store / export / verify-heads tombstone count excludes it.
- CLI verify-chain table prints Chain from + Pre-cutoff rows.
Tests: AuditLogHashChainCutoffTests (7, MSSQL), 6 verifier unit tests,
service/CLI/management wire tests; 9 mutations all killed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.
- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
the first sample after any (re)connect, triggers a probe of the other endpoint
(short-lived session: connect, read, dispose). Switch only when it reports a
Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
the previous ServiceLevel switch. Both degraded, other unreachable or level
unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
(enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
migration. Central UI: fields in the Backup card next to Failover Retry Count.
Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Re-verified #105 item 2 on the real write paths: ManagementActor bulk attribute
overrides (CLI set-overrides / management API) and InstanceService.SetAttributeOverrideAsync
(Central UI single editor) both match the template member ordinally and refuse a
case-variant name, so nothing is stored for the flattener to drop. Native-source
overrides already validate against the flattener's own output.
- New theory tests drive each path, feed whatever it stored into the real
FlatteningService, and assert every stored override is applied (exact-case row is
the positive control). Mutation: accepting case variants (OrdinalIgnoreCase) fails
them with "stored ... but the flattener dropped it as unresolved".
- The did-you-mean hint moves to TemplateEngine.OverrideNameMatching and is now also
on the UI single-attribute path; ambiguous matches get no hint.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every CLI option that can carry a secret gets a `--<name>-file <path|->`
sibling ('-' reads stdin), so the value stays out of the process list and
shell history: --password-file (global), --primary-config-file /
--backup-config-file (data-connection create|update), --auth-config-file
(external-system create|update), --connection-string-file (db-connection
create|update), --credentials-file (notification smtp create|update) and
--auth-token-file (notification sms create|update). Follows the existing
--config-file convention; `@path` is avoided because System.CommandLine
expands @-tokens as response files.
The value is read at parse time; a missing/unreadable/empty source, or '-'
on a terminal stdin, is a parse error naming option + path only, never the
content. Inline and file forms are mutually exclusive, at most one option
may read stdin, and one trailing line break is stripped. Inline forms keep
working.
docker/seed-sites.sh and docker-env2/seed-sites.sh now write the MxGateway
config (windev gateway key) to a mode-600 temp file removed by an EXIT trap
and pass --primary-config-file; seed-flow-outbound-demo.sh pipes its rig
connection string via --connection-string-file -. CLI README (new "Secrets
from a file or stdin" section + per-command tables), Component-CLI.md and
both docker READMEs updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The first mutation (instance deploy) to a site pair whose sticky node was down
returned OUTCOME_UNKNOWN: SitePairChannelProvider.ExecuteAsync treated every
Unavailable on a non-retry-safe command as possibly-applied unless the server
sent the not-dispatched trailer, flipped stickiness, and threw. The retry then
succeeded on the live peer.
A connect-phase failure provably never sent the request, so it is now treated
like the pre-dispatch rejection: GrpcRetrySafety.WasNeverSent recognises
Grpc.Net.Client's subchannel connect failure ("Error connecting to subchannel."
with a SocketException cause) and an HttpRequestException name-resolution /
connection error with a connect-only socket error. Anything that may have been
delivered (reset, premature end, GOAWAY, server Unavailable) stays
OUTCOME_UNKNOWN and is not replayed. The site->central reconcile path uses the
same ProvablyNotApplied rule.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.
The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.
Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.
Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OPCFoundation.NetStandard.Opc.Ua.Client and the test-only .Server move
1.5.378.182 -> 2.0.1, the version OtOpcUa master runs (lmxopcua#782; #799
and #801 closed). Transitive Core/Core.Types/Types/Configuration/
Security.Certificates resolve to 2.0.1.
API breaks fixed in the DCL adapter:
- Built-in types are immutable structs (Variant, DataValue, NodeId,
LocalizedText, StatusCode, ArrayOf<T>). New OpcUaValueBoundary restores
the SDK 1.5 CLR shapes for every read, subscription and alarm-event
value (byte[], UTC DateTime with no-timestamp = MinValue, Guid,
detached arrays) and builds write variants with SDK 1.5's
new Variant(object) semantics (byte[] = ByteString scalar).
- StatusCodes members are no longer constants: switches/patterns over
them (verify-outcome mapping, connection-level classification,
BadResourceUnavailable) match by value.
- CertificateIdentifier no longer caches a certificate and the
CertificateValidator/CertificateValidation event is gone: the
configuration's CertificateManager owns trust lists and the loaded
application certificate and is disposed with the session; auto-accept
and the verify probe's capture-and-reject use AcceptError.
- new UserIdentity(X509Certificate2) is gone: X509 user identities use
X509IdentityTokenHandler with an on-demand PFX provider (still fails
fast on a missing file/key).
- DefaultSessionFactory takes the telemetry context; collections are
ArrayOf<T>; EventFilter select clauses and ContentFilter operands are
built immutably.
UserName identity (lmxopcua#880): SDK 2's default handler sends a
password over 64 bytes - every 65-character OtOpcUa API key - as an
RsaEncryptedSecret that SDK 1.5 servers reject with BadIdentityTokenInvalid
and SDK 2 servers accept only within a 5-minute clock window (#873). The
DCL now always uses LegacyRsaUserNameIdentityTokenHandler (ported from
OtOpcUa be28aa44), i.e. exactly what it sent on SDK 1.5, so it logs in to
both server generations during a mixed rollout.
Tests: the in-process secured test server is ported to SDK 2 and can
admit an API key; new tests cover the handler (mutation-checked: 5 of 7
fail with the SDK default handler), the value boundary, and a 65-char key
end to end over None/Sign/SignAndEncrypt with a wrong key reaching the
key check (BadIdentityTokenRejected).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A node frozen past failure detection + StableAfter, or both sides of a partition
that heals after each downed the other, used to stay a second active node until an
operator restarted one: the peer quarantines the removed UID and Akka.Cluster
1.5.62 never acts on ThisActorSystemQuarantinedEvent.
ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via
ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for
leader-auto-down only) watches two signals:
- S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer
is still a live member here, only this side can be shut out -> restart now.
- S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still
runs the incarnation this node removed. A healed partition yields only S2, on
both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then
tiebreak.
Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower
stays and restarts only if still split one grace later. Removed peers are probed
(Identify, 5s -> 60s back-off) since nothing else contacts them after mutual
removal. Restart = Cluster.Down(self) -> existing self-down path.
The Host now exits 75 after an unplanned ActorSystem termination (and sets it on
the Windows service before it reports stopped): SCM recovery with failureflag 1
only restarts a service that stops with a non-zero code, and the watchdog used
to exit 0. Shipped default strategy stays auto-down.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.2.0 -> 0.3.0. ScadaBridgeTelemetryConfig is now
the one place both the AddZbTelemetry registration and the pre-host AddZbSerilog
logger fill their options, so ScadaBridge:Telemetry:Exporter=Otlp sends logs to the
same collector as metrics and traces, with the same Resource. The pre-host container
has no AddZbTelemetry registration for the library's log sink to follow, which is
why the exporter is passed to the logger explicitly. site.id / node.role are
unchanged (still ScadaBridge:Node:SiteId ?? central / ScadaBridge:Node:Role, applied
last so a Telemetry-section key cannot make them diverge from the log enrichers);
service.version now defaults to the entry assembly's informational version.
Host.Tests 699/699 incl. ScadaBridgeTelemetryConfigTests.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Auth 0.5.0 (all five packages) adds the passwordless ILdapGroupLookup. A new
singleton DirectoryGroupRefresher asks it for a session's current groups once
the new zb:lastgrouprefresh anchor (stamped at login) is older than
ScadaBridge:Security:LdapGroupRefresh:IntervalMinutes (default 15, Enabled
default true, CacheSeconds 60 per-node answer reuse). CookieSessionValidator
replaces the stored groups and re-maps roles; this reaches HTTP and live
circuits (once per interval, independent of the 10 s circuit refresh).
Failure policy: directory unavailable keeps the stored groups (warn, all paths
incl. fail-closed mutation checks); user not found / disabled / ambiguous /
no groups ends the session. Inert under BindMode=User (NotSupported),
LDAP disabled, or DisableLogin.
Auth 0.4.4 ApiKeyFailure.StoreUnavailable now answers 503 SERVICE_UNAVAILABLE
on the inbound API instead of a misleading 401.
Docs: Component-Security.md, components/Security.md, Component-Host.md,
register row 23 closed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Switch ScadaBridge from its own LoggerConfigurationFactory to AddZbSerilog
(ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.2.0) using the two 0.2.0 hooks:
- MinimumLevelGovernor: ScadaBridge:Logging:MinimumLevel, applied after
ReadFrom.Configuration so it outranks Serilog:MinimumLevel (never null, so the
floor never defers to the Serilog key; Override:* namespace entries still apply;
the one-shot conflict / unparseable-value warnings are unchanged).
- ConfigureLogger: the standby active-node exclusion filter (register row 71),
the configured node identity (the shared bootstrap stamps NodeHostname from
the machine name and skips empty values; an overwriting enricher keeps the
configured ScadaBridge:Node values verbatim), Enrich.FromLogContext and the
ExecutionLogCaptureSink.
ScadaBridgeLogging runs AddZbSerilog on a small pre-host builder and Program
installs the result as the static Log.Logger; both web hosts keep UseSerilog()
over it. One logger and one file-sink handle per process, usable before the
host is built (startup banner, warnings, pre-host Fatal), by SerilogAkkaLogger,
and after the host is disposed (shutdown line, unexpected-termination Fatal) —
AddZbSerilog's own DI registration would leave Log.Logger alone and have the
host dispose the logger.
LoggerConfigurationFactory is deleted (nothing else used it). Tests: existing
logging/standby/sink tests now run through the new pipeline, plus
ScadaBridgeLoggingParityTests (level precedence, Override:* still applies,
one-shot warning, configured identity incl. NodeHostname != MachineName and empty
SiteId, exact property set of a plain line, trace_id/span_id). Mutation-checked:
governor deferring, filter removed, identity enricher removed, FromLogContext
removed, capture sink removed each fail tests.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
AlarmConditionState.Severity mixed two conventions: OPC UA rows carried Part 9
Severity (higher = more severe) while MxGateway native alarms and computed
alarms carried a raw AVEVA-style priority (1 = most urgent). Every consumer
ranks higher as worse, so Worst Severity, the severity filter/sort and the
site event-log Error threshold picked the LEAST urgent MxGateway/computed
alarms.
- AlarmPriorityBands: the OtOpcUa GalaxyAlarmPriority table (lmxopcua#887,
b61f3e05), value for value: 1-250 -> 1000, 251-500 -> 750, 501-750 -> 500,
751-999 -> 250; <= 0 clamps to Critical, > 999 to Low.
- MxGatewayAlarmMapper bands the gateway severity (= AVEVA priority);
AlarmConditionStateFactory.ForComputed bands PriorityLevel. OPC UA unchanged.
- AlarmConditionState gains trailing Priority (int?): the raw source number,
exactly the pre-#129 Severity. AlarmStateChanged.Priority and the wire
AlarmStateUpdate.priority keep that meaning (no proto change); central derives
Severity by kind (SeverityFromPriority), so an older site decodes correctly.
- ScriptAlarm gains trailing Priority. The native MES 900-999 band and the MES
API Severity/MinSeverity/MaxSeverity key off Priority; rig seed scripts moved
from a.Severity to a.Priority, so MES output is unchanged.
- AlarmActor raise logs Error when the banded severity >= 700 (same threshold
as NativeAlarmActor): priority 1-500 Error, 501+ Warning (was raw >= 700).
- NativeAlarmActor upgrades pre-#129 native_alarm_state rows on rehydrate.
- Debug snapshot decode restores Condition.Priority from the row's priority.
- Docs: Commons, DCL, SiteRuntime, Communication, CentralUI, InboundAPI,
HighLevelReqs, MES alarm-status plan, CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Bump ZB.MOM.WW.Theme 0.5.0 -> 0.5.3, set ThemeShell ScrollNav=true, drop
NavMenu's own .rail-scroll wrapper and the site.css rail scroll block
(keeps .rail-updates*). NavMenu test now asserts no local wrapper.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
RealMxGatewayClient.RunEventLoopAsync stored every streamed MxEvent's
WorkerSequence as its StreamEvents resume cursor before filtering by family.
The gateway's control sentinels (ReplayGap on a resume that hit evicted
events; the opt-in StreamAttached ack) carry worker_sequence 0, so a gap
reset the cursor to 0 and the next resume became a fresh stream: no replay,
no gap signal, silent loss.
Cursor rule: advance-only (max), from real worker events (sequence > 0) only.
A ReplayGap anchors the cursor at oldest_available_sequence - 1 (the gateway's
documented gap-free resume point, valid for an empty ring too) and is logged
as a warning naming the lost interval. Sentinels never reach the value
callback.
The loop body is split into internal RunEventLoopCoreAsync(IAsyncEnumerable)
so tests drive a scripted stream; 4 of the 7 new tests fail on the old loop.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
`dotnet ef migrations script --idempotent` from migration 0 stopped at
AddSiteIdToDataConnections (Msg 207 'SiteId') and, past it, hit Msg 156/178/137
on MigrateParametersToJsonSchema's CREATE FUNCTIONs and Msg 207 on the
ExecutionId / ParentExecutionId filtered indexes. Same root cause as #126: the
script compiles a whole migration as one batch behind IF ... BEGIN guards.
(The Msg 1911 'Category' / later @legacy errors in the issue were cascades of
those failures and disappear once they are fixed.)
- New MigrationSql.Deferred(sql): wraps a body in EXEC(N'...') with quotes
doubled, so it compiles at execution time as its own batch. Migrate() runs
the identical text.
- Deferred: AddSiteIdToDataConnections step 3 (data move + cursor, DECLAREs kept
in the same body); MigrateParametersToJsonSchema's three CREATE FUNCTIONs and
the UPDATE that calls them; AddAuditLogExecutionId / AddAuditLogParentExecutionId
filtered partition-aligned indexes. No model/snapshot change.
- IdempotentMigrationScriptTests: new IdempotentScript_FromEmpty_RunsClean_AndMatchesMigrate
runs the full script on an empty DB batch by batch, re-runs it as a no-op, and
compares a schema snapshot (columns, defaults, indexes incl. filter + data space,
FKs, checks, modules, principals, permissions, partitioning, per-table row
count + checksum) against a Migrate()-built DB. Fails on the unfixed
migrations with Msg 207 'SiteId'. Batch failures now name the migration.
Schema equivalence: Migrate() before vs after this change, and the fixed
from-empty script, all produce byte-identical snapshots (939 lines); the
57->head script still runs clean and lands on the same snapshot; the seeded
data path through AddSiteIdToDataConnections + MigrateParametersToJsonSchema
gives identical rows via the fixed script and the unfixed Migrate().
Co-Authored-By: Claude <noreply@anthropic.com>
AuditMaintenanceGrants (migration 64) landed after the flip and adds
GRANT UPDATE (SourceNode) to the purger, but the tests still migrated "back
up" only to the flip and asserted the pre-64 grant list, so the class failed
in either test order (the grant missing after DownUp, or unexpected before).
Restore to head and compare the down state against the pre-64 grant set.
Co-Authored-By: Claude <noreply@anthropic.com>
`dotnet ef migrations script --idempotent` puts each migration in ONE batch
behind IF NOT EXISTS guards, and SQL Server compiles the whole batch before any
guard runs. AddAuditLogHashChain adds ChainSeq and then creates two filtered
indexes whose WHERE clause references it in the same batch, so the script
failed with Msg 207 "Invalid column name 'ChainSeq'" on any DB at migration
<= 60 (the 2026-11-11 vd03 deploy hand-inserted GO separators instead).
- AddAuditLogHashChain: the two CREATE INDEX statements go through EXEC(N'...')
so they compile at execution time. Same indexes, same snapshot; Migrate() /
database update unaffected; DBs that already ran the migration never run it
again. The column-level GRANT compiles fine (checked at execution).
- Audit of 58-64: no other migration hits the pattern (the idempotent script
from 57 now runs clean end to end; AddConfigGraphRowVersions' plain CREATE
INDEX on a new column is not compile-checked).
- docs/plans/sql/AddAuditLogHashChain.sql: same EXEC wrap. It, and
PrefixSearchIndexes.sql / AddConfigGraphRowVersions.sql, also carried a
stray UTF-8 BOM after the prepended header comment that made sqlcmd fail
with Msg 102 on line 1 of the batch; removed.
- New IdempotentMigrationScriptTests: migrates the fixture DB to 57, runs the
generated idempotent script batch by batch, asserts every migration applied,
then re-runs it as a no-op. Fails with Msg 207 without the migration fix.
Co-Authored-By: Claude <noreply@anthropic.com>
#122: the Notification Outbox KPI snapshots ORed every status in one
aggregation, which scanned the clustered index once the table held ~1 M
Parked rows (0.7-2.6 s; Health dashboard up to 5 s). Each snapshot now runs
three status-equality legs that seek their own index (~0.12 s); a cold Health
load on the rig is 0.12-0.52 s.
#123: site_events event_type/severity indexes are now composites with
timestamp, so a filtered page is read in order and stops at the limit
(severity=Error over 1 GB: 2,287 -> 15 ms cold, 115 -> 1.6 ms warm). Costs
454 -> 530 B/row, ~14 % fewer rows under the same cap. The schema apply drops
the old indexes in place; the cap size model samples timestamp three times
and re-scores inside its band with unchanged constants.
#124: the set-based AuditLog insert's NOT EXISTS probe scanned the EventId
index and concurrent writers deadlocked on S/IX page locks. WITH (FORCESEEK):
4 writers 775 rows/s + 76 deadlocks in 30 s -> 5,214 rows/s + 0. New
concurrent-writer regression test fails 4/4 without the hint.
New ResidualBench tool (eventlog, flatten, audit, audit-contention) and
docker/measure-ui-responsiveness.sh. All four residuals pass; results in
docs/plans/2026-11-10-wp4-residual-scope-results.md. Findings filed as
#122, #123 and #124 (fixed in the following commit).
The site's DataConnectionManagerActor handed VerifyEndpointAsync a NullLogger. A secured
probe is usually what first creates a node's application certificate, so the creation
Warning carrying the thumbprint every server must trust was silently discarded (found on the
rig live gate). The actor takes an optional probe logger; the Host supplies one. Test pins
creation logged once (Warning, thumbprint), not on reuse.
Sign/SignAndEncrypt sessions failed before activation with "ApplicationCertificate
for the security profile ... cannot be found" (BadConfigurationError): both
RealOpcUaClient.ConnectAsync and the verify probe built an empty CertificateIdentifier.
- New OpcUaApplicationIdentity builds the client ApplicationConfiguration for both
paths. Secured modes load the application certificate from a configurable own
store (subject + ApplicationUri), auto-creating a self-signed RSA-2048 cert only
when none exists. A persisted cert is reused across reconnect/restart and never
silently rotated: one that no longer validates (e.g. ApplicationUri mismatch)
fails closed. First-time creation is serialised per own store and logged at
Warning with the thumbprint. None mode is unchanged and never touches the store.
- New OpcUaGlobalOptions: ApplicationUri, ApplicationCertificateSubject,
OwnCertificateStorePath, AutoCreateApplicationCertificate,
ApplicationCertificateLifetimeMonths (validated).
- Owned resources: the private-key cert is disposed on disconnect, failed connect
and at probe end; sessions are now disposed after close, and a reconnect on the
same instance closes the previous session first.
- CertStoreActor wrote trusted server certs to the store ROOT, which the SDK never
reads (it uses {store}/certs), so an operator Trust never affected validation and
SDK-rejected certs never listed. It now uses certs/ and migrates legacy root files
on start.
- Rig: all site nodes (docker/, docker-env2/) put the four PKI stores on the
persistent /app/data volume with a pinned ApplicationUri.
- Tests: in-process strict-trust StandardServer (test-only Opc.Ua.Server 1.5.378.106)
exercising the real adapter through ConnectAsync and VerifyEndpointAsync for Sign
and SignAndEncrypt: allow + read + reconnect + restart on one identity, deny when
the client doesn't trust the server (BadCertificateUntrusted), deny when the server
doesn't trust the client (BadSecurityChecksFailed), None untouched, concurrent
provisioning creates one cert, URI mismatch fails without rotating, auto-create off.
Against the pre-fix client these tests fail with the exact #121 error.
#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.
#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
registration is declared inside the launcher's and executor's try, so it ends before the catch
blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.
Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.
Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Script_UsingDatabase_QueriesViaGateway failed in about 2 of 27 full InboundAPI runs. Looping the
suite 30 times captured the failure 4 times, always as "Internal script error". The exception
underneath was SqliteException 'no such table: Machine'.
Cause: the seeding helper opened its keep-alive SqliteConnection in a local. Once the helper
returned, nothing referenced that connection. After a GC (likely during the Roslyn compile, under
full-suite load), Microsoft.Data.Sqlite's pool reclaimed the leaked connection and closed the last
connection to the shared in-memory database, which destroyed it.
Fix: hold the keep-alive in a static field. InboundDatabaseHelperTests had the same latent bug and
gets the same fix. The test now forces GC plus SqliteConnection.ClearAllPools() before the query.
That makes the old pattern fail deterministically and the fix pass deterministically.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LogContext is an AsyncLocal and does not cross an Akka message hop, so the run's id now rides the
message: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) on
SetStaticAttributeCommand, AttributeValueChanged (not for an inferred writer), ScriptCallRequest and
the four RouteTo*Requests, plus a new additive SourceExecutionId on GetAttributeRequest,
WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest and WriteTagBatchRequest.
ScriptRuntimeContext stamps its id on the requests it sends, the Instance Actor hands it on to the
DCL, and the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in
AroundReceive (ExecutionLogContext.PushFor: one type test when the message names no execution). The
inbound executor pushes the request's ExecutionId for the method execution.
Tests (TestKit): Instance Actor line carries the message's id, no leak from the sender's ambient
context, scope ends with the message, per-write ids inside a batched SetAttributes, ids forwarded on
DCL writes, ScriptRuntimeContext stamps; DCL adapter lines on single/batch writes carry the id;
inbound executor lines carry the request id; Commons mapping + no serialized member. Each mutation-
proved. Still open on #56: a queryable per-execution sink and a Flow Debugger section. The DCL
actors' own Akka _log lines bypass Serilog (filed #120).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
CollisionDetector counted a derived template's own override row (same name as a
parent member, IsInherited=false) as a second definition alongside the
parent's row, so AddAttribute/AddAlarm/AddScript for a fresh name failed with a
false "Naming collision" once any override existed. The inheritance walk now
keeps a (type, name) shadow set: a member defined at a nearer level of the
chain shadows the same member further up (effective member set, by name),
including overrides at mid levels. Real collisions (e.g. against a composed
module) still report.
Regression tests in TemplateEngine.Tests (detector + service, attribute/alarm/
script, mid-level override, positive control); 6 fail without the fix.
The #118 importer's additions-before-overrides ordering is now unnecessary but
harmless and was left in place.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Bundles did not record that a template was a derived slot template, so the
importer re-created every slot as a BASE template named after the slot.
- Export: TemplateDto gains an additive, null-omitted `Derivation`
(owner path + base path by stable names); a derived template carries only
its own rows (overrides/additions, not inherited placeholders). The closure
pulls in each derived template's base/owner; base orders first.
schemaVersion 1.1 -> 1.2, bundleFormatVersion stays 1.
- Import: base templates go through the ordinary passes; derived templates are
ordered topologically (DerivedTemplatePlan; cycle / duplicate / malformed =
preview Blocker + SemanticValidationException before anything is staged),
their owner and base resolved by path (#100 root rule), and the slot is
created with TemplateService.AddCompositionAsync, synced in place when it
already exists with the same base, or deleted+recomposed when the base
differs; own rows applied as overrides, then ResyncInheritedMembersAsync.
Missing owner/base fails that record (BundleImportDerivedTemplateUnresolved
audit + ImportResult warning), never creating a base in its place.
- Preview/diff: derived rows are keyed by qualified name and compared against
the target slot at the same owner path (own rows + base), so an unchanged
slot is Identical and a same-named base template is never matched.
- Pre-#118 bundles import exactly as before (new fixture
pre-118-derived-plaintext.scadabundle, written by a13aabc7). The field is in
content, pinned by the signed manifest's contentHash.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
MapStatusCode returned Uncertain for any non-zero non-Bad code, so GoodLocalOverride,
GoodClamped and info-bit Good variants degraded to Uncertain. Now top two bits decide:
00 Good, 01 Uncertain, 10/11 Bad (BadResourceUnavailable stays Bad, #112 untouched).
The SDK IsGood/IsUncertain(uint) helpers were tried first and failed the subcode tests,
so an explicit mask is used. MxGateway mapper uses the OPC-DA quality byte (>=192 Good)
and is not affected.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every downing decision of LeaderAutoDowningProvider is now logged under a
[leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable),
unreachable observed + its window (Warning), skipped because already Down/Exiting,
pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept
pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own
AutoDown logged only the final Down. Decision semantics are unchanged.
New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests),
built from production BuildHocon with self-first seeds:
- both nodes cold-start together -> one cluster, nobody downed past FD + window;
- lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up,
then the peer cold-starts and joins;
- flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the
window, repeated past it in total -> never downed; positive control asserts the cuts
really produced unreachability;
- partition held past the window -> each side downs the other, both stay Up and
leader (the accepted dual-active trade);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider,
only leader-auto-down loads the in-repo provider.
Unit: flapping never downs; a flapping member that finally stays unreachable is downed
exactly once a full window after its LAST unreachable; each decision log line.
Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable
ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping
test fail; DOWNING log dropped -> log test fails; provider never downs -> partition
test fails; auto-down miswired to the new provider -> switch-off theory fails.
Docker failover drill not run (it would rebuild the shared scadabridge:latest image
and reuse the shared rig volumes while another agent works in the repo); filed as
deploy gate ScadaBridge#116. Default remains auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TemplateLibrary.GetIdByNameAsync was first-row-wins, so a derived slot template
could shadow a same-named base. Now: exact name, base first, derived-only name
only when unique, otherwise unresolved (composition audit says ambiguous).
Tracked-template indexes are base-first too.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
When the run reset the response flag via SetAttribute before the six-argument
handshake, wait for that reset to echo (shared timeout) before writing batch +
trigger, so the previous run's cached CompleteFlag can't satisfy it. Covers the
unported DB-owned MES/receiver scripts. Pending reset consumed per handshake.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
OtOpcUa's out-of-process script runner publishes 0x80040000 with a null value.
- OpcUaDataConnection: logs entry/recovery once per tag, names the symbolic
status in read errors; never a disconnect (not in the reconnect status set).
- InstanceActor: a Bad update with no value is a quality-only transition
(keep value, quality Bad, stream publish, no script/alarm/waiter re-eval),
matching the connection-drop rule. Bad with a value is unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
#113: AuditLogPartitionMaintenance (monthly SPLIT RANGE) and the
backfill-source-node endpoint now use the AuditLog:MaintenanceConnectionString
login when set (unset = unchanged). Migration AuditMaintenanceGrants gives
scadabridge_audit_purger ALTER ANY DATASPACE and column-level UPDATE (SourceNode).
#114: archive cleanup deletes the writer's own *.tmp files and manifest-less
data files older than 24 h inside {Directory}/{YYYY} (DeletedOrphan, dry-run aware).
#115: head-export failures bump scadabridge.audit.chain_head_export.failures and
Degrade the /health/ready entry audit-chain-head-export (failing or stale > 2 intervals).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions recorded: the archive destination stays a directory/UNC
(no object-storage client; share + permissions named at enablement),
parquet-tools-only reading (no re-import), unsigned like the chain (#107).
AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed
AuditLog:RetentionDays, counted from the archived month's end) and
DeleteSupersededStaleArchives (default true) run after every purge tick.
A retried purge's stale Archived archive is deleted only when a later,
verifying, Purged archive of the same partition provably holds every row:
same prior anchor and the stale chain-end hash at its chain-end sequence
(row or tombstone), every unchained row present by EventId; otherwise it is
reported KeptStale. A sole copy left at Archived is never touched. Deletion
removes only the archive name's own three files in its folder, manifest
first, never a manifest-supplied path. `audit archive cleanup [--dry-run]`
= CleanupAuditArchives (Administrator).
AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy,
sole copy, retention and the path guard; each safety rule mutation-checked.
Follow-ups filed: #113, #114, #115.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 1 (owner decision: no signing). AuditLog:HashChain:HeadExport (default
off): both centrals periodically read every month's chain head (LastSeq,
LastHash, purge anchor, chained rows, tombstones; one snapshot under the
chain lock SHARED) and append the changed heads to a sink the DB login cannot
write - File (append-only JSONL per node per day, local or UNC) or Syslog
(RFC 5424 over UDP / TCP octet counting). `audit verify-heads --path` reads
those records back (file or syslog-collector lines) and the new
VerifyAuditChainHeads command checks each against the database: Matches /
Extended / Purged, or Mismatch / Regressed / MissingHead / MissingLink /
CountMismatch (exit 3). AuditChainHeadStoreTests forges a chain with the
production hasher: verify-chain says Intact, verify-heads says Mismatch.
Item 3. AuditLog:MaintenanceConnectionString (${secret:}-capable, unset =
today's single login) runs the chain job, chain verify / head export, the
partition switch-out + archive and the per-channel purge on
AuditMaintenanceContext (keyed audit-maintenance repository/archive, purge
actor prefers them). AuditMaintenanceLoginTests runs the hardened setup on
SQL Server: writer (datareader/datawriter/audit_writer) ingests and is
refused chaining and deleting; the chainer+purger login does the full cycle.
Two-login grants documented in Component-AuditLog.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the
twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId)
on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred
(EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve
indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production
scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql.
The partition-switch purge staging table now reads the live clustered key
order from sys.index_columns, so it switches either shape (old binaries
hard-code the old order: ship with the binaries, both centrals stopped).
Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward
clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14;
FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the
staging key (mutation: hard-coded old key fails SWITCH) and the read ratio.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions on #110: one signing key per environment; TrustedKeys stay hand-edited;
Signing:ProductionBoundary (new, default off) logs a startup warning and reports a Degraded
bundle-signing-policy /health/ready entry while RequireSignedBundles is off; the private key
is read via ${secret:} (proved end to end, never logged); cluster pull decided not to be
built; differential deletions stay deferred. Adds docs/operations/bundle-signing-runbook.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. PendingUpdatesEvaluator re-flattened every
deployed instance on each cache-TTL expiry even when nothing had changed.
- Migration AddConfigGraphRowVersions: a non-concurrency rowversion column
+ index on the 18 configuration tables the evaluation reads (Templates +
5 member tables, Instances + 4 override/binding tables, DataConnections,
DeployedConfigSnapshots, Sites, SharedScripts, ExternalSystemDefinitions,
ExternalSystemMethods, DatabaseConnectionDefinitions). SQL Server only in
the model; SQLite test contexts are untouched. Script:
docs/plans/sql/AddConfigGraphRowVersions.sql. Joins the coordinated
central-pair migration window (#107 item 4).
- IConfigGraphWatermarkReader / ConfigGraphWatermarkReader: ONE query of
MAX(RowVersion) + COUNT_BIG(*) per table (count catches deletes, which
raise no rowversion). rowversion is engine-bumped for any writer, so a
peer central node's edit moves it — not the per-process watermark MG-3
removed.
- The secret store (outside the DB) gets its own stamp: SHA-256 over
ListAsync(includeDeleted: true) metadata.
- Evaluator re-serves the last CLEAN snapshot while watermark + secret
stamp are unchanged; every PendingUpdatesFullEvaluationEvery'th (default
10, validated >= 1, 1 = off) is full regardless — the safety net for a
same-KEK re-wrap or an input table missing from the set. Always full when
the token is unavailable/unreadable, after an evaluation that could not
compute a site's artifact hash, on Re-check, and after a deploy completion.
It only re-serves a real earlier answer, never manufactures one.
Tests: SQL Server (real migrated DB) — token stable without writes, moves
for a peer-connection INSERT/UPDATE and a non-max DELETE, moves for a local
EF write, a detached Update still saves (not a concurrency token), and a
guard that every table the evaluator's repository reads touch is covered.
Evaluator — reuse without flatten (re-served answer still shows the stale
instance), watermark move, 4 secret changes, every-Nth, 5 no-token modes,
unclean evaluation not reused, cache Re-check and deploy completion force
full. Mutation-proven: token compare, clean check, cache force flag, row
count, and an uncovered read each turn tests red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. An inbound API method has no site
affinity, so the F2 route-time scope only bounded routed SITE traffic;
the method's central-side effects (Database.*, ExternalSystem,
notifications) still ran for a scoped Deployer. FlowDebugRunner now
refuses FlowDebugRunMethodCommand for a site-scoped caller (site ids,
not Administrator) before the script executes, audits the refusal as a
failed InboundRequest under the operator's name, and raises
SiteScopeViolationException -> ManagementUnauthorized, so the Central
UI and the CLI (same /management command) share the gate.
The page disables the "Inbound API method" entry for scoped Deployers
with the reason (tooltip + inline note), starts them on script entry,
and ignores ?methodId= / a stored method choice. Replay and the
instance-scoped script / value-change Live runs are unchanged. The
route-time scope is kept as defence in depth.
Tests: scoped refused (in- and out-of-scope routing; script never runs,
refusal audited), unscoped/Administrator allowed, scoped Replay allowed,
scoped in-scope value-change Live still runs; bUnit for the disabled
entry, the drill-in, and the unscoped control. Mutation-proven (gate
and UI flag reverted -> 4 red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Exports can now carry a detached ECDSA P-256 signature over the exact
manifest.json bytes in a new optional manifest.sig zip entry. The manifest
already pins the content hash, so a verified signature covers the whole
bundle, including the plaintext source environment the confirm step relies
on. Off by default (ScadaBridge:Transport:Signing): with the defaults,
exports are unsigned and imports accept signed and unsigned bundles.
- The bundle format does not change: bundleFormatVersion stays 1,
manifest.json and the AES-GCM associated data are untouched, and older
importers open entries by name and ignore manifest.sig. Three entries fit
within the default MaxBundleEntryCount of 4.
- Import policy runs after the content-hash check and before any passphrase
work. A signature that fails to verify against a trusted key is always
refused. RequireSignedBundles additionally refuses unsigned bundles and
bundles signed by an untrusted key. A refusal is BundleSignatureException
(Commons), shown on the UI upload step and on CLI preview/import.
- The signature status, key id and signer are recorded on BundleSession and
shown in the import wizard. They are also returned on PreviewBundleResult
and recorded on the BundleImported audit row. BundleExported records
Signed/SigningKeyId.
- Key material is pluggable. IBundleSigningKeySource and IBundleTrustStore
are TryAdd'd; the defaults read Signing:PrivateKey (meant to be a
${secret:} reference) and Signing:TrustedKeys. Key ids are SPKI
fingerprints. Startup validation rejects unparseable or non-P-256 keys.
- Compatibility proof: the checked-in fixtures pre-signing-plaintext and
pre-signing-encrypted were exported by origin/main 3ff463b6, before this
change, and are imported end to end with signing off and with signing
configured.
The design leaves key-per-env vs per-org, trust distribution and rotation,
and the production default for RequireSignedBundles open. These are listed
in Component-Transport.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
GrpcPullAuditEventsInvoker, GrpcPullSiteCallsInvoker and
GrpcPullTraceEventsInvoker called WithSiteCredentials without the endpoint,
so they composed the PSK over insecure channel credentials and threw at
channel construction for any https site address. They now pass the endpoint
(same change as 9138f3ed): https composes over SecureSsl, http is unchanged.
An internal ConfigureHandler seam lets tests trust a generated certificate;
production leaves it null.
Tests: real Kestrel HTTP/2-over-TLS with the real ControlPlaneAuthInterceptor
for all three pulls (correct PSK accepted, wrong PSK PermissionDenied,
untrusted certificate fails). Reverting the fix turns all five red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Central re-bound its HTTP surface from ASPNETCORE_URLS keeping only the
port, so https://+:443 silently served plaintext HTTP. The parser now keeps
the scheme (ParseHttpBindEndpoints); an https entry binds UseHttps with the
node certificate from the new ScadaBridge:Node:Tls section (CertificatePath
+ CertificatePassword, or CertificateThumbprint + StoreName + StoreLocation;
default unset) and the node refuses to boot, with a clear error, when https
is requested without a certificate. http entries bind exactly as before.
Tests: scheme parsing, refuse-to-boot, http endpoint config identical to the
legacy inline binding (reflected effective Kestrel listen options), and a
real Kestrel HTTPS listener with a generated .pfx serving h2 and HTTP/1.1 by
ALPN while plaintext to the TLS port fails. Reverting the fix turns the
refuse-to-boot and real-listener tests red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
GET /api/audit/export?format=parquet returned 501. It now streams the same 24
columns as csv/jsonl (AuditExportRow, CsvHeader order) as one Parquet file,
one 10,000-row row group at a time. Each row group is encoded into a small
non-seekable staging buffer and copied to the response asynchronously, because
Kestrel rejects the synchronous writes the Parquet writer issues.
New management commands, with CLI parity:
- ListAuditArchives -> `audit archive list` (AuditArchiveListResult: enabled
flag + every manifest).
- VerifyAuditArchive(Name) -> `audit archive verify --name` (files, chain,
purge anchor); exit 0 verified, 3 not verified, 1 error, 2 unauthorized.
Both carry verify-chain's gate: OperationalAudit roles (Administrator, Viewer),
site-scoped non-Administrator refused because an archive spans every site.
VerifyAuditArchive uses the long-running Ask timeout. Error codes
ARCHIVE_NOT_CONFIGURED, INVALID_ARGUMENT, NOT_FOUND.
RequiredRoleMatrixTests freezes both gates; ManagementSurfaceParityTests now
probes `audit archive list|verify` alongside `audit verify-chain`.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.
- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
tombstone file = the month's per-channel purge tombstones. Buffered row
groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
chain heads and tombstones in one transaction under the chain lock SHARED,
writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
.tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
month's head and prior anchor) via tmp + rename, manifest last. Archives are
never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
overload. Under the EXCLUSIVE chain lock it re-counts the partition and
re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
both still equal the archive's, so the switch's checkpoint anchor is exactly
the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
from the archived rows and tombstones (AuditChainVerifier), and the chain end
against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
absolute; RowGroupSize, timeouts bounded.
No migration: the purger role already reads AuditLog, the head and tombstones.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
WithSiteCredentials always composed the per-site PSK over
ChannelCredentials.Insecure. Grpc.Net.Client refuses that on an https
address ("insecure channel credentials and can't use a HttpClient with a
'https' scheme"), so every https control-plane endpoint threw at channel
construction - in SitePairChannelProvider the throw was caught and the
site address logged as malformed, i.e. a TLS site was silently
undialable. The old comment claiming TLS needed no change here was wrong.
WithSiteCredentials now takes the endpoint: https composes over
ChannelCredentials.SecureSsl (PSK retained inside TLS, no insecure
opt-in); anything else, including callers that omit it, keeps the
previous composition unchanged. SitePairChannelProvider,
SiteStreamGrpcClient and CentralChannelProvider pass their endpoint. The
three AuditLog pull clients are deliberately left for after #42.
Tests: real Kestrel HTTP/2-over-TLS (ALPN) with the real
ControlPlaneAuthInterceptor - correct PSK accepted, wrong PSK still
PermissionDenied, untrusted server cert fails with an
AuthenticationException cause, the pre-fix composition throws on https,
http composition unchanged; plus each production builder accepts https
(mutation check: disabling the TLS branch turns those red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ScadaBridge:Cluster:RemotingTls (ClusterRemotingTlsOptions) renders
akka.remote.dot-netty.tcp enable-ssl + ssl { certificate, mutual auth,
hostname validation } only when Enabled. Chain validation is always on
(no suppress-validation switch). Certificate is a .pfx path + password
(${secret:} reference) or an OS-store thumbprint.
ClusterOptionsValidator, TLS on only: exactly one certificate source, a
legal store location, and akka.ssl.tcp:// on every seed - Akka.Remote
advertises akka.ssl.tcp addresses once SSL is on, so an akka.tcp seed is
never answered and the join retries forever with no startup error.
ClusterBootstrapGuard.SeedPattern accepted only one scheme segment, so
akka.ssl.tcp seeds parsed as nothing and the guard would silently never
engage on a TLS pair; it now accepts any number.
Default HOCON is byte-identical: 12 rendered documents (3 downing
strategies x guard on/off x Central/Site) captured before the change and
diffed after, all identical; a permanent test pins that a disabled block
with every field set renders the default document. A real ActorSystem
booted on the emitted TLS document comes up as akka.ssl.tcp (mutation
check: forcing enable-ssl=false turns it red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET
1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the
leader among the reachable members downs a peer unreachable for StableAfter; a
non-leader records it as pending and downs it on becoming leader; reachable-again
or removed members are forgotten; Down/Exiting members are skipped. Its window is
carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no
deprecated auto-down-unreachable-after key is emitted and Akka's boot-time
"keep-majority will be used instead" warning goes away. DownRemovalMargin reads
akka.cluster.down-removal-margin exactly as AutoDowning does (default off).
New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon
and ClusterOptionsValidator. The default stays "auto-down"; auto-down and
keep-oldest HOCON is byte-identical to before. Placed in Host, not
ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays
Akka-free).
Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec
cases (leader-only, pending-on-leader-change, losing leadership, reachable again,
removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests
clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node
cluster plus a test that Akka actually loaded the provider and its downing actor.
Mutation-checked: a provider that never calls Cluster.Down fails both failover
tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed
mutations each fail the unit tests.
Remaining gate before switching the default: the docker failover drill
(DRILL_MODE=active and standby mode) on leader-auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
verify-chain was a no-op stub that printed "not enabled" and exited 0 without
contacting the server. It now sends VerifyAuditChain(Month) over
POST /management and maps the verdict to the exit code: 0 intact or not
chained, 3 broken (first break: partition, ChainSeq, EventId, kind), 1 error,
2 unauthorized. Table and JSON renderings.
ManagementActor: new HandleVerifyAuditChain arm, gated on the OperationalAudit
roles (Administrator, Viewer) and refusing a site-scoped caller because a
month's chain spans every site; INVALID_ARGUMENT for a bad month, CHAIN_BUSY
when the chain lock is held; long-running Ask timeout. Result enums serialise
as names.
Parity: RequiredRoleMatrixTests freezes the gate; ManagementSurfaceParityTests
now probes `audit verify-chain` (the one audit leaf that is a management
contract) instead of skipping the whole audit group.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Joseph Doherty