LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor (#140) stays.
GracefulStopWithOpenSyncStreamTests' positive control passed the stop
in ~900 ms under 0.3.8, because the library alone now ends the stream.
The control registers LocalDbSyncService without the host lifetime (the
0.3.7 behaviour), so it still reproduces the 8 s drain; verified it
fails on 0.3.8 without that change.
Release build 0 warnings. Host.Tests 733/733, IntegrationTests 133/133
(incl. the LocalDb site-pair convergence tests), Communication.Tests
843/843, HealthMonitoring.Tests 131/131, SiteEventLogging.Tests 118/118.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A graceful stop of a site node whose peer holds an open LocalDb sync stream
left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906,
scadaproj#57). WebApplication stops the web server before AkkaHostedService,
whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited
the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7
observes only the call token, which Kestrel cancels at the END of the drain.
CancelAllStreams only covered SiteStreamGrpcServer's own streams.
- AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs
before any hosted service's StopAsync (Kestrel's included): marks the stop
intentional, raises ApplicationStopping (so CancelAllStreams still precedes
actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason),
bounded by the host shutdown token. StopAsync returns the same completed
shutdown. Exit codes unchanged: planned stop exits 0; only a termination
outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting
here, so no ClusterLeavingReason/-1 exit-code path is involved.
- ShutdownStreamCancellationInterceptor, innermost on both the central and the
site AddGrpc pipelines: every streaming call's token is linked with
ApplicationStopping; new streams are refused Unavailable while stopping.
Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned.
Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real
WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor
+ a real LocalDb SyncBackgroundService peer + the production AkkaHostedService
forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp
the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms;
after 960 ms stop, member removed at 950 ms, drain began at 959 ms.
Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor
passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync
does not raise ApplicationStopping; lifecycle interface dropped; site pipeline
loses the interceptor.
Release build of the solution: 0 warnings. Host.Tests 733/733,
IntegrationTests 133/133, Communication.Tests 843/843.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Owner decision on #107 (2026-11-13): enabling AuditLog:HashChain must not
chain the historical backlog. main chained every unchained row oldest-first.
- The first chaining pass on a pristine database (no head, no chained row)
writes a chain-start anchor: AuditLogChainTombstone (cutoff month,
ChainSeq 0), PurgedAtUtc = DB SYSUTCDATETIME(), RowHash =
SHA-256("ScadaBridge.AuditLog.ChainCutoff.v1" || YYYY-MM || ticks).
Written once under the exclusive chain lock, so both centrals agree
whatever their clocks/config. No migration: existing table + grants.
- Only rows with OccurredAtUtc >= cutoff are chained. The anchor hash is
the cutoff month's genesis, binding the cutoff into the chain.
- AuditLog:HashChain:ChainBacklog (default false) restores chain-everything,
read only on the very first enablement. Legacy chains (heads, no anchor)
keep chaining everything.
- verify-chain splits unchained rows into UnchainedRows vs PreCutoffRows
(+ ChainFromUtc); new breaks ChainedBeforeCutoff (row re-dated before the
cutoff, or cutoff moved forward) and CutoffAnchorInvalid (anchor hash /
month / multiplicity). Archive verify uses the anchor from its own
tombstone file or the DB.
- Partition switch-out keeps the anchor (deletes ChainSeq > 0 only); the
head store / export / verify-heads tombstone count excludes it.
- CLI verify-chain table prints Chain from + Pre-cutoff rows.
Tests: AuditLogHashChainCutoffTests (7, MSSQL), 6 verifier unit tests,
service/CLI/management wire tests; 9 mutations all killed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.
- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
the first sample after any (re)connect, triggers a probe of the other endpoint
(short-lived session: connect, read, dispose). Switch only when it reports a
Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
the previous ServiceLevel switch. Both degraded, other unreachable or level
unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
(enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
migration. Central UI: fields in the Backup card next to Failover Retry Count.
Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Re-verified #105 item 2 on the real write paths: ManagementActor bulk attribute
overrides (CLI set-overrides / management API) and InstanceService.SetAttributeOverrideAsync
(Central UI single editor) both match the template member ordinally and refuse a
case-variant name, so nothing is stored for the flattener to drop. Native-source
overrides already validate against the flattener's own output.
- New theory tests drive each path, feed whatever it stored into the real
FlatteningService, and assert every stored override is applied (exact-case row is
the positive control). Mutation: accepting case variants (OrdinalIgnoreCase) fails
them with "stored ... but the flattener dropped it as unresolved".
- The did-you-mean hint moves to TemplateEngine.OverrideNameMatching and is now also
on the UI single-attribute path; ambiguous matches get no hint.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Every CLI option that can carry a secret gets a `--<name>-file <path|->`
sibling ('-' reads stdin), so the value stays out of the process list and
shell history: --password-file (global), --primary-config-file /
--backup-config-file (data-connection create|update), --auth-config-file
(external-system create|update), --connection-string-file (db-connection
create|update), --credentials-file (notification smtp create|update) and
--auth-token-file (notification sms create|update). Follows the existing
--config-file convention; `@path` is avoided because System.CommandLine
expands @-tokens as response files.
The value is read at parse time; a missing/unreadable/empty source, or '-'
on a terminal stdin, is a parse error naming option + path only, never the
content. Inline and file forms are mutually exclusive, at most one option
may read stdin, and one trailing line break is stripped. Inline forms keep
working.
docker/seed-sites.sh and docker-env2/seed-sites.sh now write the MxGateway
config (windev gateway key) to a mode-600 temp file removed by an EXIT trap
and pass --primary-config-file; seed-flow-outbound-demo.sh pipes its rig
connection string via --connection-string-file -. CLI README (new "Secrets
from a file or stdin" section + per-command tables), Component-CLI.md and
both docker READMEs updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The channel-pair log templates used {SiteId}, which collides with the
node's own enriched SiteId property, so the failover warning read
"Site central node A unavailable...". Renamed to {TargetSiteId} in all
six templates in SitePairChannelProvider.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The first mutation (instance deploy) to a site pair whose sticky node was down
returned OUTCOME_UNKNOWN: SitePairChannelProvider.ExecuteAsync treated every
Unavailable on a non-retry-safe command as possibly-applied unless the server
sent the not-dispatched trailer, flipped stickiness, and threw. The retry then
succeeded on the live peer.
A connect-phase failure provably never sent the request, so it is now treated
like the pre-dispatch rejection: GrpcRetrySafety.WasNeverSent recognises
Grpc.Net.Client's subchannel connect failure ("Error connecting to subchannel."
with a SocketException cause) and an HttpRequestException name-resolution /
connection error with a connect-only socket error. Anything that may have been
delivered (reset, premature end, GOAWAY, server Unavailable) stays
OUTCOME_UNKNOWN and is not replayed. The site->central reconcile path uses the
same ProvablyNotApplied rule.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.
The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.
Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.
Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
docker/seed-sites.sh and docker-env2/seed-sites.sh hard-coded the windev
gateway API key. They now read MXGW_APIKEY from the environment, or from
a private file (default ~/.config/scadabridge/windev-mxgw-apikey,
override MXGW_APIKEY_FILE). With no key the MxGateway connection step is
skipped and the rest of the seed runs. READMEs note where to put it.
The key remains in git history (2 commits), so it should be rotated on
the windev gateway.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Unreplicated, the site-x survivor came up empty (by design: no LocalDb peer);
replicated, kill and freeze each lost only the 26-30 s takeover window, every
buffered message was delivered once, and the 4 duplicate ticks came from the
frozen victim's 6 s zombie window. Central kill: 0 gap.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Unreplicated, the standby holds no deployed configs, data connections or
S&F buffer: a seeded active kill showed the survivor take over the cluster
in 28 s but load no instances and run no scripts. Dev-only sync key.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Adds an env2-owned HTTP sink container (scadabridge-env2-sfsink) whose 503
mode makes ExternalSystem.CachedCall buffer, and an EnvSoak template whose 2 s
SoakTick sends one tick-numbered cached call, so loss and duplicates are
countable at the sink.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- docker/build.sh: SCADABRIDGE_IMAGE picks the tag (default scadabridge:latest).
- docker-env2/docker-compose.yml: node image is ${SCADABRIDGE_IMAGE:-scadabridge:latest},
so env2 can run its own build without retagging the primary rig's image.
- docker-env2/docker-compose.leader-auto-down.yml: opt-in overlay for both env2 pairs,
mirroring docker/docker-compose.leader-auto-down.yml. Shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_ENV=primary|env2 (container prefix, Traefik :9100,
pair site-x).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Read-only export of DEV vd03 and QA vq03 (2026-11-12): the only
Children["DelmiaRecv"] user is CvdReactor.ProcessRecipeDownload (script 4,
identical on both envs; DEV 3 / QA 9 instances), and the only ScriptAlarm
.Severity reader is DEV TestMesMachine.AlarmStatus (script 42, SBTEST01/02).
QA has no #131 work.
Adds the original (rollback) and fixed bodies, a guarded check/apply/rollback
script that re-sends every script field, and the apply plan: #52 any time,
#131 with or after the dd4f1c2a binary. The old #131 body still compiles on
main but returns banded values; the fixed body is what fails to compile on the
deployed binaries. Rig evidence for both gates is in the README.
Refs #52#131
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
OPCFoundation.NetStandard.Opc.Ua.Client and the test-only .Server move
1.5.378.182 -> 2.0.1, the version OtOpcUa master runs (lmxopcua#782; #799
and #801 closed). Transitive Core/Core.Types/Types/Configuration/
Security.Certificates resolve to 2.0.1.
API breaks fixed in the DCL adapter:
- Built-in types are immutable structs (Variant, DataValue, NodeId,
LocalizedText, StatusCode, ArrayOf<T>). New OpcUaValueBoundary restores
the SDK 1.5 CLR shapes for every read, subscription and alarm-event
value (byte[], UTC DateTime with no-timestamp = MinValue, Guid,
detached arrays) and builds write variants with SDK 1.5's
new Variant(object) semantics (byte[] = ByteString scalar).
- StatusCodes members are no longer constants: switches/patterns over
them (verify-outcome mapping, connection-level classification,
BadResourceUnavailable) match by value.
- CertificateIdentifier no longer caches a certificate and the
CertificateValidator/CertificateValidation event is gone: the
configuration's CertificateManager owns trust lists and the loaded
application certificate and is disposed with the session; auto-accept
and the verify probe's capture-and-reject use AcceptError.
- new UserIdentity(X509Certificate2) is gone: X509 user identities use
X509IdentityTokenHandler with an on-demand PFX provider (still fails
fast on a missing file/key).
- DefaultSessionFactory takes the telemetry context; collections are
ArrayOf<T>; EventFilter select clauses and ContentFilter operands are
built immutably.
UserName identity (lmxopcua#880): SDK 2's default handler sends a
password over 64 bytes - every 65-character OtOpcUa API key - as an
RsaEncryptedSecret that SDK 1.5 servers reject with BadIdentityTokenInvalid
and SDK 2 servers accept only within a 5-minute clock window (#873). The
DCL now always uses LegacyRsaUserNameIdentityTokenHandler (ported from
OtOpcUa be28aa44), i.e. exactly what it sent on SDK 1.5, so it logs in to
both server generations during a mixed rollout.
Tests: the in-process secured test server is ported to SDK 2 and can
admit an API key; new tests cover the handler (mutation-checked: 5 of 7
fail with the SDK default handler), the value boundary, and a 65-char key
end to end over None/Sign/SignAndEncrypt with a wrong key reaching the
key check (BadIdentityTokenRejected).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
0.3.1 stops the gRPC/HttpClient instrumentation tracing telemetry's own
OTLP export calls (the Serilog log sink's LogsService/Export spans).
The app's own outgoing gRPC/HTTP spans are unchanged.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
.dockerignore keeps .git/ out of the build context, so the in-container
publish had no repository and AssemblyInformationalVersion (service.version,
VersionInfo) was a bare 1.0.0. docker/build.sh (used by docker/deploy.sh and
docker-env2/deploy.sh) now passes git rev-parse HEAD, suffixed -dirty for an
uncommitted tree, and the Dockerfile hands it to dotnet publish as
SourceRevisionId. Without the arg the build still succeeds as 1.0.0. The ARG
sits after the source COPY so a new commit does not bust the restore layer.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
A node frozen past failure detection + StableAfter, or both sides of a partition
that heals after each downed the other, used to stay a second active node until an
operator restarted one: the peer quarantines the removed UID and Akka.Cluster
1.5.62 never acts on ThisActorSystemQuarantinedEvent.
ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via
ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for
leader-auto-down only) watches two signals:
- S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer
is still a live member here, only this side can be shut out -> restart now.
- S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still
runs the incarnation this node removed. A healed partition yields only S2, on
both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then
tiebreak.
Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower
stays and restarts only if still split one grace later. Removed peers are probed
(Identify, 5s -> 60s back-off) since nothing else contacts them after mutual
removal. Restart = Cluster.Down(self) -> existing self-down path.
The Host now exits 75 after an unplanned ActorSystem termination (and sets it on
the Windows service before it reports stopped): SCM recovery with failureflag 1
only restarts a service that stops with a non-zero code, and the watchdog used
to exit 0. Shipped default strategy stays auto-down.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.Telemetry(.Serilog) 0.2.0 -> 0.3.0. ScadaBridgeTelemetryConfig is now
the one place both the AddZbTelemetry registration and the pre-host AddZbSerilog
logger fill their options, so ScadaBridge:Telemetry:Exporter=Otlp sends logs to the
same collector as metrics and traces, with the same Resource. The pre-host container
has no AddZbTelemetry registration for the library's log sink to follow, which is
why the exporter is passed to the logger explicitly. site.id / node.role are
unchanged (still ScadaBridge:Node:SiteId ?? central / ScadaBridge:Node:Role, applied
last so a Telemetry-section key cannot make them diverge from the log enrichers);
service.version now defaults to the entry assembly's informational version.
Host.Tests 699/699 incl. ScadaBridgeTelemetryConfigTests.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Auth 0.5.0 (all five packages) adds the passwordless ILdapGroupLookup. A new
singleton DirectoryGroupRefresher asks it for a session's current groups once
the new zb:lastgrouprefresh anchor (stamped at login) is older than
ScadaBridge:Security:LdapGroupRefresh:IntervalMinutes (default 15, Enabled
default true, CacheSeconds 60 per-node answer reuse). CookieSessionValidator
replaces the stored groups and re-maps roles; this reaches HTTP and live
circuits (once per interval, independent of the 10 s circuit refresh).
Failure policy: directory unavailable keeps the stored groups (warn, all paths
incl. fail-closed mutation checks); user not found / disabled / ambiguous /
no groups ends the session. Inert under BindMode=User (NotSupported),
LDAP disabled, or DisableLogin.
Auth 0.4.4 ApiKeyFailure.StoreUnavailable now answers 503 SERVICE_UNAVAILABLE
on the inbound API instead of a misleading 401.
Docs: Component-Security.md, components/Security.md, Component-Host.md,
register row 23 closed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.
Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.
- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
(kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.
Refs #116, #106, #132
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Switch ScadaBridge from its own LoggerConfigurationFactory to AddZbSerilog
(ZB.MOM.WW.Telemetry(.Serilog) 0.1.0 -> 0.2.0) using the two 0.2.0 hooks:
- MinimumLevelGovernor: ScadaBridge:Logging:MinimumLevel, applied after
ReadFrom.Configuration so it outranks Serilog:MinimumLevel (never null, so the
floor never defers to the Serilog key; Override:* namespace entries still apply;
the one-shot conflict / unparseable-value warnings are unchanged).
- ConfigureLogger: the standby active-node exclusion filter (register row 71),
the configured node identity (the shared bootstrap stamps NodeHostname from
the machine name and skips empty values; an overwriting enricher keeps the
configured ScadaBridge:Node values verbatim), Enrich.FromLogContext and the
ExecutionLogCaptureSink.
ScadaBridgeLogging runs AddZbSerilog on a small pre-host builder and Program
installs the result as the static Log.Logger; both web hosts keep UseSerilog()
over it. One logger and one file-sink handle per process, usable before the
host is built (startup banner, warnings, pre-host Fatal), by SerilogAkkaLogger,
and after the host is disposed (shutdown line, unexpected-termination Fatal) —
AddZbSerilog's own DI registration would leave Log.Logger alone and have the
host dispose the logger.
LoggerConfigurationFactory is deleted (nothing else used it). Tests: existing
logging/standby/sink tests now run through the new pipeline, plus
ScadaBridgeLoggingParityTests (level precedence, Override:* still applies,
one-shot warning, configured identity incl. NodeHostname != MachineName and empty
SiteId, exact property set of a plain line, trace_id/span_id). Mutation-checked:
governor deferring, filter removed, identity enricher removed, FromLogContext
removed, capture sink removed each fail tests.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
AlarmConditionState.Severity mixed two conventions: OPC UA rows carried Part 9
Severity (higher = more severe) while MxGateway native alarms and computed
alarms carried a raw AVEVA-style priority (1 = most urgent). Every consumer
ranks higher as worse, so Worst Severity, the severity filter/sort and the
site event-log Error threshold picked the LEAST urgent MxGateway/computed
alarms.
- AlarmPriorityBands: the OtOpcUa GalaxyAlarmPriority table (lmxopcua#887,
b61f3e05), value for value: 1-250 -> 1000, 251-500 -> 750, 501-750 -> 500,
751-999 -> 250; <= 0 clamps to Critical, > 999 to Low.
- MxGatewayAlarmMapper bands the gateway severity (= AVEVA priority);
AlarmConditionStateFactory.ForComputed bands PriorityLevel. OPC UA unchanged.
- AlarmConditionState gains trailing Priority (int?): the raw source number,
exactly the pre-#129 Severity. AlarmStateChanged.Priority and the wire
AlarmStateUpdate.priority keep that meaning (no proto change); central derives
Severity by kind (SeverityFromPriority), so an older site decodes correctly.
- ScriptAlarm gains trailing Priority. The native MES 900-999 band and the MES
API Severity/MinSeverity/MaxSeverity key off Priority; rig seed scripts moved
from a.Severity to a.Priority, so MES output is unchanged.
- AlarmActor raise logs Error when the banded severity >= 700 (same threshold
as NativeAlarmActor): priority 1-500 Error, 501+ Warning (was raw >= 700).
- NativeAlarmActor upgrades pre-#129 native_alarm_state rows on rehydrate.
- Debug snapshot decode restores Condition.Priority from the row's priority.
- Docs: Commons, DCL, SiteRuntime, Communication, CentralUI, InboundAPI,
HighLevelReqs, MES alarm-status plan, CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Client (DataConnectionLayer) and the test-only Server (DataConnectionLayer.Tests)
move together to 1.5.378.182, the last SDK 1.5 build OtOpcUa ran before 2.0.1.
Patch-level hop with no API changes and no source edits. Transitive Core,
Configuration, Security.Certificates and Types resolve to .182 under CPM
transitive pinning, and existing pins already satisfy its floors.
Since .156, an issuer CA must carry the keyCertSign KeyUsage, including the CA of
a server cert the DCL trusts. SDK-generated self-signed identities are unaffected,
and the strict-trust secure-connect tests pass unchanged.
The 2.0.x move is deferred until lmxopcua#799 and #801 close.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Pins ZB.MOM.WW.LocalDb, .Replication and .Contracts at 0.3.7 (published
2026-11-12). Wire-compatible; no config or code change needed.
- 0.3.6 (scadaproj#36): the library sync service enforces
LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
its refuse-every-stream-when-blank rule; its doc comment is updated. The
rig's site-a pair already carries one identical key on both nodes.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
node offline past TombstoneRetention no longer resurrects deleted rows.
The survivor's ledger (sf_messages / site_events deletes) grows for the
length of the outage; optional TombstoneHardCap bounds it.
Deploy notes added to the topology + installation guides, the S&F
requirement and CLAUDE.md: stop/start the pair together, same ApiKey on
both nodes, ledger growth while a peer is down.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Bump ZB.MOM.WW.Theme 0.5.0 -> 0.5.3, set ThemeShell ScrollNav=true, drop
NavMenu's own .rail-scroll wrapper and the site.css rail scroll block
(keeps .rail-updates*). NavMenu test now asserts no local wrapper.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
RealMxGatewayClient.RunEventLoopAsync stored every streamed MxEvent's
WorkerSequence as its StreamEvents resume cursor before filtering by family.
The gateway's control sentinels (ReplayGap on a resume that hit evicted
events; the opt-in StreamAttached ack) carry worker_sequence 0, so a gap
reset the cursor to 0 and the next resume became a fresh stream: no replay,
no gap signal, silent loss.
Cursor rule: advance-only (max), from real worker events (sequence > 0) only.
A ReplayGap anchors the cursor at oldest_available_sequence - 1 (the gateway's
documented gap-free resume point, valid for an empty ring too) and is logged
as a warning naming the lost interval. Sentinels never reach the value
callback.
The loop body is split into internal RunEventLoopCoreAsync(IAsyncEnumerable)
so tests drive a scripted stream; 4 of the 7 new tests fail on the old loop.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
`dotnet ef migrations script --idempotent` from migration 0 stopped at
AddSiteIdToDataConnections (Msg 207 'SiteId') and, past it, hit Msg 156/178/137
on MigrateParametersToJsonSchema's CREATE FUNCTIONs and Msg 207 on the
ExecutionId / ParentExecutionId filtered indexes. Same root cause as #126: the
script compiles a whole migration as one batch behind IF ... BEGIN guards.
(The Msg 1911 'Category' / later @legacy errors in the issue were cascades of
those failures and disappear once they are fixed.)
- New MigrationSql.Deferred(sql): wraps a body in EXEC(N'...') with quotes
doubled, so it compiles at execution time as its own batch. Migrate() runs
the identical text.
- Deferred: AddSiteIdToDataConnections step 3 (data move + cursor, DECLAREs kept
in the same body); MigrateParametersToJsonSchema's three CREATE FUNCTIONs and
the UPDATE that calls them; AddAuditLogExecutionId / AddAuditLogParentExecutionId
filtered partition-aligned indexes. No model/snapshot change.
- IdempotentMigrationScriptTests: new IdempotentScript_FromEmpty_RunsClean_AndMatchesMigrate
runs the full script on an empty DB batch by batch, re-runs it as a no-op, and
compares a schema snapshot (columns, defaults, indexes incl. filter + data space,
FKs, checks, modules, principals, permissions, partitioning, per-table row
count + checksum) against a Migrate()-built DB. Fails on the unfixed
migrations with Msg 207 'SiteId'. Batch failures now name the migration.
Schema equivalence: Migrate() before vs after this change, and the fixed
from-empty script, all produce byte-identical snapshots (939 lines); the
57->head script still runs clean and lands on the same snapshot; the seeded
data path through AddSiteIdToDataConnections + MigrateParametersToJsonSchema
gives identical rows via the fixed script and the unfixed Migrate().
Co-Authored-By: Claude <noreply@anthropic.com>
AuditMaintenanceGrants (migration 64) landed after the flip and adds
GRANT UPDATE (SourceNode) to the purger, but the tests still migrated "back
up" only to the flip and asserted the pre-64 grant list, so the class failed
in either test order (the grant missing after DownUp, or unexpected before).
Restore to head and compare the down state against the pre-64 grant set.
Co-Authored-By: Claude <noreply@anthropic.com>
`dotnet ef migrations script --idempotent` puts each migration in ONE batch
behind IF NOT EXISTS guards, and SQL Server compiles the whole batch before any
guard runs. AddAuditLogHashChain adds ChainSeq and then creates two filtered
indexes whose WHERE clause references it in the same batch, so the script
failed with Msg 207 "Invalid column name 'ChainSeq'" on any DB at migration
<= 60 (the 2026-11-11 vd03 deploy hand-inserted GO separators instead).
- AddAuditLogHashChain: the two CREATE INDEX statements go through EXEC(N'...')
so they compile at execution time. Same indexes, same snapshot; Migrate() /
database update unaffected; DBs that already ran the migration never run it
again. The column-level GRANT compiles fine (checked at execution).
- Audit of 58-64: no other migration hits the pattern (the idempotent script
from 57 now runs clean end to end; AddConfigGraphRowVersions' plain CREATE
INDEX on a new column is not compile-checked).
- docs/plans/sql/AddAuditLogHashChain.sql: same EXEC wrap. It, and
PrefixSearchIndexes.sql / AddConfigGraphRowVersions.sql, also carried a
stray UTF-8 BOM after the prepended header comment that made sqlcmd fail
with Msg 102 on line 1 of the batch; removed.
- New IdempotentMigrationScriptTests: migrates the fixture DB to 57, runs the
generated idempotent script batch by batch, asserts every migration applied,
then re-runs it as a no-op. Fails with Msg 207 without the migration fix.
Co-Authored-By: Claude <noreply@anthropic.com>
#122: the Notification Outbox KPI snapshots ORed every status in one
aggregation, which scanned the clustered index once the table held ~1 M
Parked rows (0.7-2.6 s; Health dashboard up to 5 s). Each snapshot now runs
three status-equality legs that seek their own index (~0.12 s); a cold Health
load on the rig is 0.12-0.52 s.
#123: site_events event_type/severity indexes are now composites with
timestamp, so a filtered page is read in order and stops at the limit
(severity=Error over 1 GB: 2,287 -> 15 ms cold, 115 -> 1.6 ms warm). Costs
454 -> 530 B/row, ~14 % fewer rows under the same cap. The schema apply drops
the old indexes in place; the cap size model samples timestamp three times
and re-scores inside its band with unchanged constants.
#124: the set-based AuditLog insert's NOT EXISTS probe scanned the EventId
index and concurrent writers deadlocked on S/IX page locks. WITH (FORCESEEK):
4 writers 775 rows/s + 76 deadlocks in 30 s -> 5,214 rows/s + 0. New
concurrent-writer regression test fails 4/4 without the hint.
New ResidualBench tool (eventlog, flatten, audit, audit-contention) and
docker/measure-ui-responsiveness.sh. All four residuals pass; results in
docs/plans/2026-11-10-wp4-residual-scope-results.md. Findings filed as
#122, #123 and #124 (fixed in the following commit).
vd03 is the DEV box (vd = DEV, vq03/vq04 = QA). Historical docs called it prod/production.
Only vd03-referential uses are changed; generic 'production' wording (production HOCON,
production code paths, ASPNETCORE_ENVIRONMENT=Production, real-world production guidance)
is unchanged. One verbatim git commit message quoted in PLAN-07 is left as recorded.
The site's DataConnectionManagerActor handed VerifyEndpointAsync a NullLogger. A secured
probe is usually what first creates a node's application certificate, so the creation
Warning carrying the thumbprint every server must trust was silently discarded (found on the
rig live gate). The actor takes an optional probe logger; the Host supplies one. Test pins
creation logged once (Warning, thumbprint), not on reuse.
Sign/SignAndEncrypt sessions failed before activation with "ApplicationCertificate
for the security profile ... cannot be found" (BadConfigurationError): both
RealOpcUaClient.ConnectAsync and the verify probe built an empty CertificateIdentifier.
- New OpcUaApplicationIdentity builds the client ApplicationConfiguration for both
paths. Secured modes load the application certificate from a configurable own
store (subject + ApplicationUri), auto-creating a self-signed RSA-2048 cert only
when none exists. A persisted cert is reused across reconnect/restart and never
silently rotated: one that no longer validates (e.g. ApplicationUri mismatch)
fails closed. First-time creation is serialised per own store and logged at
Warning with the thumbprint. None mode is unchanged and never touches the store.
- New OpcUaGlobalOptions: ApplicationUri, ApplicationCertificateSubject,
OwnCertificateStorePath, AutoCreateApplicationCertificate,
ApplicationCertificateLifetimeMonths (validated).
- Owned resources: the private-key cert is disposed on disconnect, failed connect
and at probe end; sessions are now disposed after close, and a reconnect on the
same instance closes the previous session first.
- CertStoreActor wrote trusted server certs to the store ROOT, which the SDK never
reads (it uses {store}/certs), so an operator Trust never affected validation and
SDK-rejected certs never listed. It now uses certs/ and migrates legacy root files
on start.
- Rig: all site nodes (docker/, docker-env2/) put the four PKI stores on the
persistent /app/data volume with a pinned ApplicationUri.
- Tests: in-process strict-trust StandardServer (test-only Opc.Ua.Server 1.5.378.106)
exercising the real adapter through ConnectAsync and VerifyEndpointAsync for Sign
and SignAndEncrypt: allow + read + reconnect + restart on one identity, deny when
the client doesn't trust the server (BadCertificateUntrusted), deny when the server
doesn't trust the client (BadSecurityChecksFailed), None untouched, concurrent
provisioning creates one cert, URI mismatch fails without rotating, auto-create off.
Against the pre-fix client these tests fail with the exact #121 error.
#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.
#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
registration is declared inside the launcher's and executor's try, so it ends before the catch
blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.
Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.
Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Script_UsingDatabase_QueriesViaGateway failed in about 2 of 27 full InboundAPI runs. Looping the
suite 30 times captured the failure 4 times, always as "Internal script error". The exception
underneath was SqliteException 'no such table: Machine'.
Cause: the seeding helper opened its keep-alive SqliteConnection in a local. Once the helper
returned, nothing referenced that connection. After a GC (likely during the Roslyn compile, under
full-suite load), Microsoft.Data.Sqlite's pool reclaimed the leaked connection and closed the last
connection to the shared in-memory database, which destroyed it.
Fix: hold the keep-alive in a static field. InboundDatabaseHelperTests had the same latent bug and
gets the same fix. The test now forces GC plus SqliteConnection.ClearAllPools() before the query.
That makes the old pattern fail deterministically and the fix pass deterministically.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LogContext is an AsyncLocal and does not cross an Akka message hop, so the run's id now rides the
message: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) on
SetStaticAttributeCommand, AttributeValueChanged (not for an inferred writer), ScriptCallRequest and
the four RouteTo*Requests, plus a new additive SourceExecutionId on GetAttributeRequest,
WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest and WriteTagBatchRequest.
ScriptRuntimeContext stamps its id on the requests it sends, the Instance Actor hands it on to the
DCL, and the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in
AroundReceive (ExecutionLogContext.PushFor: one type test when the message names no execution). The
inbound executor pushes the request's ExecutionId for the method execution.
Tests (TestKit): Instance Actor line carries the message's id, no leak from the sender's ambient
context, scope ends with the message, per-write ids inside a batched SetAttributes, ids forwarded on
DCL writes, ScriptRuntimeContext stamps; DCL adapter lines on single/batch writes carry the id;
inbound executor lines carry the request id; Commons mapping + no serialized member. Each mutation-
proved. Still open on #56: a queryable per-execution sink and a Flow Debugger section. The DCL
actors' own Akka _log lines bypass Serilog (filed #120).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
CollisionDetector counted a derived template's own override row (same name as a
parent member, IsInherited=false) as a second definition alongside the
parent's row, so AddAttribute/AddAlarm/AddScript for a fresh name failed with a
false "Naming collision" once any override existed. The inheritance walk now
keeps a (type, name) shadow set: a member defined at a nearer level of the
chain shadows the same member further up (effective member set, by name),
including overrides at mid levels. Real collisions (e.g. against a composed
module) still report.
Regression tests in TemplateEngine.Tests (detector + service, attribute/alarm/
script, mid-level override, positive control); 6 fail without the fix.
The #118 importer's additions-before-overrides ordering is now unnecessary but
harmless and was left in place.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Bundles did not record that a template was a derived slot template, so the
importer re-created every slot as a BASE template named after the slot.
- Export: TemplateDto gains an additive, null-omitted `Derivation`
(owner path + base path by stable names); a derived template carries only
its own rows (overrides/additions, not inherited placeholders). The closure
pulls in each derived template's base/owner; base orders first.
schemaVersion 1.1 -> 1.2, bundleFormatVersion stays 1.
- Import: base templates go through the ordinary passes; derived templates are
ordered topologically (DerivedTemplatePlan; cycle / duplicate / malformed =
preview Blocker + SemanticValidationException before anything is staged),
their owner and base resolved by path (#100 root rule), and the slot is
created with TemplateService.AddCompositionAsync, synced in place when it
already exists with the same base, or deleted+recomposed when the base
differs; own rows applied as overrides, then ResyncInheritedMembersAsync.
Missing owner/base fails that record (BundleImportDerivedTemplateUnresolved
audit + ImportResult warning), never creating a base in its place.
- Preview/diff: derived rows are keyed by qualified name and compared against
the target slot at the same owner path (own rows + base), so an unchanged
slot is Identical and a same-named base template is never matched.
- Pre-#118 bundles import exactly as before (new fixture
pre-118-derived-plaintext.scadabundle, written by a13aabc7). The field is in
content, pinned by the signed manifest's contentHash.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
MapStatusCode returned Uncertain for any non-zero non-Bad code, so GoodLocalOverride,
GoodClamped and info-bit Good variants degraded to Uncertain. Now top two bits decide:
00 Good, 01 Uncertain, 10/11 Bad (BadResourceUnavailable stays Bad, #112 untouched).
The SDK IsGood/IsUncertain(uint) helpers were tried first and failed the subcode tests,
so an explicit mask is used. MxGateway mapper uses the OPC-DA quality byte (>=192 Good)
and is not affected.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every downing decision of LeaderAutoDowningProvider is now logged under a
[leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable),
unreachable observed + its window (Warning), skipped because already Down/Exiting,
pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept
pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own
AutoDown logged only the final Down. Decision semantics are unchanged.
New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests),
built from production BuildHocon with self-first seeds:
- both nodes cold-start together -> one cluster, nobody downed past FD + window;
- lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up,
then the peer cold-starts and joins;
- flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the
window, repeated past it in total -> never downed; positive control asserts the cuts
really produced unreachability;
- partition held past the window -> each side downs the other, both stay Up and
leader (the accepted dual-active trade);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider,
only leader-auto-down loads the in-repo provider.
Unit: flapping never downs; a flapping member that finally stays unreachable is downed
exactly once a full window after its LAST unreachable; each decision log line.
Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable
ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping
test fail; DOWNING log dropped -> log test fails; provider never downs -> partition
test fails; auto-down miswired to the new provider -> switch-off theory fails.
Docker failover drill not run (it would rebuild the shared scadabridge:latest image
and reuse the shared rig volumes while another agent works in the repo); filed as
deploy gate ScadaBridge#116. Default remains auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Joseph Doherty