#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.
#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
registration is declared inside the launcher's and executor's try, so it ends before the catch
blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.
Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.
Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LogContext is an AsyncLocal and does not cross an Akka message hop, so the run's id now rides the
message: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) on
SetStaticAttributeCommand, AttributeValueChanged (not for an inferred writer), ScriptCallRequest and
the four RouteTo*Requests, plus a new additive SourceExecutionId on GetAttributeRequest,
WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest and WriteTagBatchRequest.
ScriptRuntimeContext stamps its id on the requests it sends, the Instance Actor hands it on to the
DCL, and the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in
AroundReceive (ExecutionLogContext.PushFor: one type test when the message names no execution). The
inbound executor pushes the request's ExecutionId for the method execution.
Tests (TestKit): Instance Actor line carries the message's id, no leak from the sender's ambient
context, scope ends with the message, per-write ids inside a batched SetAttributes, ids forwarded on
DCL writes, ScriptRuntimeContext stamps; DCL adapter lines on single/batch writes carry the id;
inbound executor lines carry the request id; Commons mapping + no serialized member. Each mutation-
proved. Still open on #56: a queryable per-execution sink and a Flow Debugger section. The DCL
actors' own Akka _log lines bypass Serilog (filed #120).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Every downing decision of LeaderAutoDowningProvider is now logged under a
[leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable),
unreachable observed + its window (Warning), skipped because already Down/Exiting,
pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept
pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own
AutoDown logged only the final Down. Decision semantics are unchanged.
New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests),
built from production BuildHocon with self-first seeds:
- both nodes cold-start together -> one cluster, nobody downed past FD + window;
- lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up,
then the peer cold-starts and joins;
- flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the
window, repeated past it in total -> never downed; positive control asserts the cuts
really produced unreachability;
- partition held past the window -> each side downs the other, both stay Up and
leader (the accepted dual-active trade);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider,
only leader-auto-down loads the in-repo provider.
Unit: flapping never downs; a flapping member that finally stays unreachable is downed
exactly once a full window after its LAST unreachable; each decision log line.
Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable
ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping
test fail; DOWNING log dropped -> log test fails; provider never downs -> partition
test fails; auto-down miswired to the new provider -> switch-off theory fails.
Docker failover drill not run (it would rebuild the shared scadabridge:latest image
and reuse the shared rig volumes while another agent works in the repo); filed as
deploy gate ScadaBridge#116. Default remains auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
When the run reset the response flag via SetAttribute before the six-argument
handshake, wait for that reset to echo (shared timeout) before writing batch +
trigger, so the previous run's cached CompleteFlag can't satisfy it. Covers the
unported DB-owned MES/receiver scripts. Pending reset consumed per handshake.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the
twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId)
on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred
(EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve
indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production
scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql.
The partition-switch purge staging table now reads the live clustered key
order from sys.index_columns, so it switches either shape (old binaries
hard-code the old order: ship with the binaries, both centrals stopped).
Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward
clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14;
FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the
staging key (mutation: hard-coded old key fails SWITCH) and the read ratio.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions on #110: one signing key per environment; TrustedKeys stay hand-edited;
Signing:ProductionBoundary (new, default off) logs a startup warning and reports a Degraded
bundle-signing-policy /health/ready entry while RequireSignedBundles is off; the private key
is read via ${secret:} (proved end to end, never logged); cluster pull decided not to be
built; differential deletions stay deferred. Adds docs/operations/bundle-signing-runbook.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. PendingUpdatesEvaluator re-flattened every
deployed instance on each cache-TTL expiry even when nothing had changed.
- Migration AddConfigGraphRowVersions: a non-concurrency rowversion column
+ index on the 18 configuration tables the evaluation reads (Templates +
5 member tables, Instances + 4 override/binding tables, DataConnections,
DeployedConfigSnapshots, Sites, SharedScripts, ExternalSystemDefinitions,
ExternalSystemMethods, DatabaseConnectionDefinitions). SQL Server only in
the model; SQLite test contexts are untouched. Script:
docs/plans/sql/AddConfigGraphRowVersions.sql. Joins the coordinated
central-pair migration window (#107 item 4).
- IConfigGraphWatermarkReader / ConfigGraphWatermarkReader: ONE query of
MAX(RowVersion) + COUNT_BIG(*) per table (count catches deletes, which
raise no rowversion). rowversion is engine-bumped for any writer, so a
peer central node's edit moves it — not the per-process watermark MG-3
removed.
- The secret store (outside the DB) gets its own stamp: SHA-256 over
ListAsync(includeDeleted: true) metadata.
- Evaluator re-serves the last CLEAN snapshot while watermark + secret
stamp are unchanged; every PendingUpdatesFullEvaluationEvery'th (default
10, validated >= 1, 1 = off) is full regardless — the safety net for a
same-KEK re-wrap or an input table missing from the set. Always full when
the token is unavailable/unreadable, after an evaluation that could not
compute a site's artifact hash, on Re-check, and after a deploy completion.
It only re-serves a real earlier answer, never manufactures one.
Tests: SQL Server (real migrated DB) — token stable without writes, moves
for a peer-connection INSERT/UPDATE and a non-max DELETE, moves for a local
EF write, a detached Update still saves (not a concurrency token), and a
guard that every table the evaluator's repository reads touch is covered.
Evaluator — reuse without flatten (re-served answer still shows the stale
instance), watermark move, 4 secret changes, every-Nth, 5 no-token modes,
unclean evaluation not reused, cache Re-check and deploy completion force
full. Mutation-proven: token compare, clean check, cache force flag, row
count, and an uncovered read each turn tests red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decision (b), 2026-11-09. An inbound API method has no site
affinity, so the F2 route-time scope only bounded routed SITE traffic;
the method's central-side effects (Database.*, ExternalSystem,
notifications) still ran for a scoped Deployer. FlowDebugRunner now
refuses FlowDebugRunMethodCommand for a site-scoped caller (site ids,
not Administrator) before the script executes, audits the refusal as a
failed InboundRequest under the operator's name, and raises
SiteScopeViolationException -> ManagementUnauthorized, so the Central
UI and the CLI (same /management command) share the gate.
The page disables the "Inbound API method" entry for scoped Deployers
with the reason (tooltip + inline note), starts them on script entry,
and ignores ?methodId= / a stored method choice. Replay and the
instance-scoped script / value-change Live runs are unchanged. The
route-time scope is kept as defence in depth.
Tests: scoped refused (in- and out-of-scope routing; script never runs,
refusal audited), unscoped/Administrator allowed, scoped Replay allowed,
scoped in-scope value-change Live still runs; bUnit for the disabled
entry, the drill-in, and the unscoped control. Mutation-proven (gate
and UI flag reverted -> 4 red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Evaluated against the current pages and their data sources. The notification
outbox is central-owned, offset-paged and has its own status lifecycle. Site
calls are site-owned and keyset-paged, with a different lifecycle and a
Retry/Discard that relays to the owning site. A union grid would need a
two-cursor merge, a status filter that does not line up, and buttons that hide
different semantics. The revisit trigger (operator confusion reports) has not
fired.
Recommends against building it now. If confusion is reported, start with
cross-links and a combined "stuck items" Health card.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Neither feature is specified well enough to build safely, so this records the
open questions, the options and a recommendation instead of code.
- Pull: the existing CLI can already orchestrate export from one cluster and
import into another with per-cluster auth and audit. A server-side pull
needs decisions on the machine credential, source-side authorisation,
network direction and passphrase handling. Recommendation: orchestrate it
with the CLI and RequireSignedBundles; write no code now.
- Differential: imports already auto-skip Identical rows, so a full bundle
applies only what changed. The baseline, deletions and target drift are
undefined. Recommendation: keep YAGNI; if it is ever needed, use
per-entry content hashes plus --since, and never tombstones.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Per-hop current state with code evidence (browser/Traefik/Kestrel, CLI,
central<->site gRPC + PSK, LocalDb sync, Akka remoting, OPC UA /
MxGateway), the recommended mode per hop (Kestrel HTTPS + ALPN, gRPC over
TLS with validation and the PSK retained, Akka DotNetty SSL with a cert
per node and mutual auth), certificate issuance/storage/rotation, rig and
wonder impact, a staged rollout, rollback, a test plan and 13 open owner
decisions. Findings: the central ASPNETCORE_URLS re-bind drops the
scheme (an https URL binds plaintext); Akka.NET has no secure-cookie, so
mTLS is the only remoting authentication. Component-ClusterInfrastructure
documents the new RemotingTls option.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET
1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the
leader among the reachable members downs a peer unreachable for StableAfter; a
non-leader records it as pending and downs it on becoming leader; reachable-again
or removed members are forgotten; Down/Exiting members are skipped. Its window is
carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no
deprecated auto-down-unreachable-after key is emitted and Akka's boot-time
"keep-majority will be used instead" warning goes away. DownRemovalMargin reads
akka.cluster.down-removal-margin exactly as AutoDowning does (default off).
New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon
and ClusterOptionsValidator. The default stays "auto-down"; auto-down and
keep-oldest HOCON is byte-identical to before. Placed in Host, not
ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays
Akka-free).
Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec
cases (leader-only, pending-on-leader-change, losing leadership, reachable again,
removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests
clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node
cluster plus a test that Akka actually loaded the provider and its downing actor.
Mutation-checked: a provider that never calls Cluster.Down fails both failover
tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed
mutations each fail the unit tests.
Remaining gate before switching the default: the docker failover drill
(DRILL_MODE=active and standby mode) on leader-auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audit roadmap T1 (alog.md §11.4): one chain per UTC calendar month of
OccurredAtUtc. Each chained row carries ChainSeq (gap-free within its month)
and RowHash = SHA-256(prev.RowHash || canonical(row)); the v1 canonical form
(AuditChainHasher) is pinned by a known-answer vector computed independently.
- Migration AddAuditLogHashChain: nullable ChainSeq/RowHash on dbo.AuditLog
(metadata-only add), AuditLogChainHead (per-month chain end + partition-purge
checkpoint anchor), AuditLogChainTombstone, two partition-aligned filtered
indexes, scadabridge_audit_chainer role with a column-level UPDATE grant.
Production script docs/plans/sql/AddAuditLogHashChain.sql.
- AuditLogHashChain chains unchained rows oldest month first under an
exclusive sp_getapplock (zero timeout, so a dual-active central pair still
yields one chain), with an optimistic head check and a ChainSeq IS NULL
row-count check that roll a pass back instead of leaving a gap. The ingest
path is untouched.
- Purges keep the chain verifiable: the partition switch records the month's
checkpoint anchor and drops its tombstones; the per-channel DELETE writes a
tombstone per chained row in the same transaction (OUTPUT INTO). Both, and
the SourceNode backfill, hold the chain lock; the backfill skips chained
rows. The switch staging table gains the two new columns.
- AuditChainVerifier reports the first break (partition, ChainSeq, EventId,
kind) for edited, deleted, reordered rows and tail truncation.
- AuditLogHashChainService (central, default OFF via AuditLog:HashChain).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
(a) Inbound API compile state (#75) is per central node: a recompile cleared the
known-bad record only on the node that answered, and the peer kept refusing the
unchanged script with SCRIPT_COMPILE_FAILED (the per-request revision check only
notices a changed script). Create / update / delete (ManagementActor post-commit),
RecompileApiMethod, and API methods written by a Transport bundle import now publish
InboundApiMethodInvalidated on the central cache-invalidation topic (#100 item 3).
The new InboundApiCacheInvalidationHandler drops that method's compiled handler,
known-bad record and cached ApiMethod row on the peer, which compiles the current
text on its next request.
(b) docs/components/InboundAPI.md no longer claims a direct DB edit stays stale
until restart: the per-request revision check picks it up once the node's cached
row expires (ApiMethodCacheTtl, 5 s). Component-InboundAPI, Component-Host and the
#85 cache audit describe the peer invalidation.
(c) Decision: the import's locked-override warnings keep EXACT (ordinal) name
matching, because FlatteningService matches override names ordinally. An override
differing from a locked member only by case never reaches that member's lock check
(the flattener drops it as unresolved), so a "targets a LOCKED member" warning for it
would be false. The comparer is now named (LockedOverrideNameComparer) and documented,
and pinned by tests on both sides (flattener + import).
Tests: the peer's known-bad record is cleared by a recompile (and validate leaves it
alone); create/update/delete and the import publish the invalidation; the handler
drops only the named method; composition root registers the handler; case-only
override gets no lock warning; the flattener matches override names exactly.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast.
It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation,
carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated,
RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated)
in an envelope stamped with the publisher's process-unique origin id.
- Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until
the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which
skips its own node's envelopes and hands the peer's to every registered
ICentralCacheInvalidationHandler (a failing handler does not stop the rest).
Replaces SitePskInvalidationSubscriber.
- Migration of #96: SitePskRotationInvalidator publishes through the bus and is the
PSK handler; SitePskInvalidated keeps its type name and is also published bare on
site-psk-invalidation, which the subscriber still listens to, so a mixed-version
central pair keeps exchanging PSK invalidations. Remove after one release.
- Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping
invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on
the peer instead of after RoleMappingCacheSeconds.
- Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9
ManagementActor handlers and the Transport bundle importer publish
ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.)
TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and
subscriber unit tests (origin filter, legacy topic), peer-node tests through the real
caches in ManagementService.Tests, importer publish, composition root, and a real
two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests)
proving every message type crosses with the production serializer. Docs: Host,
Communication, Security, ManagementService, ESG, and the #85 cache audit.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Deployments free-text search and the config-audit EntityName filter no longer
emit LIKE '%term%' by default. Prefix (LIKE 'term%') is the default; substring
is an explicit opt-in (DeploymentSearchMode.Contains / entityNameContains)
always bounded by a DeployedAt/Timestamp window (explicit, else 30 days).
Migration PrefixSearchIndexes adds DeployedBy, RevisionHash and EntityName
indexes. Deployments page gains a 'Match anywhere' checkbox.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TemplateNativeAlarmSource.IsMesReportable + tri-state InstanceNativeAlarmSourceOverride
.IsMesReportableOverride flatten onto ResolvedNativeAlarmSource; NativeAlarmActor stamps every
mirrored condition and InstanceActor the quiet-binding placeholder, so ScriptAlarm,
Alarms.MesReportableAsync() and the #60 wire fields carry it. Additive to the 900-999 severity band.
Migration AddNativeAlarmSourceMesReportable. Management commands, CLI --mes-reportable, Central UI
(template dialog checkbox, instance override select), Transport DTOs/importer/diff and the revision
hash (true-only, so existing hashes are unchanged) carry it.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:
- the secret is written through /admin/secrets: central replaces the secrets
library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
publishes SitePskInvalidated on a new DistributedPubSub topic
(site-psk-invalidation) so the peer central node drops its copy and its
resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
WithSiteCredentials wraps every per-site channel's handler in
PskRejectionRefreshHandler, so the next call re-reads the key (covers
out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.
Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The site script and alarm on-trigger launchers push the run identity onto LogContext for the
run's lifetime and the host now enables Enrich.FromLogContext(), so log lines written on the
run's own async call chain carry the id. Bounded part only: LogContext does not cross an Akka
message hop, so Instance Actor/DCL/S&F lines and a site_events execution-id column remain open
(documented in the step-logs design doc section 9).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audit (docs/plans/2026-11-09-central-cache-staleness-audit.md) of every central in-process
cache holding configuration or secret state, with its staleness bound on the peer central
node. Every config cache with peer staleness is TTL-bounded (5-60 s) and Traefik routes to the
active node only, so no peer case was clearly a bug and no cross-node broadcast is added; the
doc names the existing DistributedPubSub mediator (health replication topic) as the mechanism
if one is wanted. Owner decisions listed: RoleMappingCache 30 s revocation lag on the peer,
SitePskProvider caching a PSK forever across an in-place rotation (both nodes), and the #83
ExternalDefinitionLookupCache 10 s bound.
Fixed: ManagementActor API-method create/update/delete invalidated only the compiled handler,
not ApiMethodCache, so the node that made the change kept serving the pre-change row for up to
the 5 s TTL (a deleted method stayed callable; a cached not-found hid a new one). The handlers
now drop the entry post-commit, like the role-mapping cache.
Tests: ApiMethodCacheInvalidationTests (create/update/delete fail without the fix; a
negative control checks other methods stay cached).
(ScadaBridge#85)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ScriptCallResult and RouteToCallResponse gain trailing Truncated/LogsTruncated,
filled from the site FlowTraceCollector after Complete(); RouteToCallResponseDto
fields 8/9 carry them (sitecommand regen). FlowDebugRunner ORs them into
FlowDebugRunResult on the script entry, and on the method entry through a new
optional truncation sink on RouteHelper.WithTraceSink (carried through every
With* clone). Previously a site run whose collector dropped rows came back as a
silently short list. Older sites send false.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Central's Debug View live stream and Alarm Summary cache always saw false
because neither wire message had the field. Additive proto fields, vendored
C# regenerated with docker/regen-proto.sh (sitestream + sitecommand), mapped
both ways in StreamRelayActor/SiteStreamGrpcClient and SiteCommandDtoMapper.
The live stream publishes Alarm Actor events, which never carried the flag,
so InstanceActor.HandleAlarmStateChanged now stamps it before retaining and
publishing. A pre-#60 site sends the proto3 default. No UI badge yet.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval
(10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and
deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the
refresh). The delete is conditional on the checked_at it read, so a node that came back
keeps its fresh row; a returning node re-writes its row on its first pass.
Also: DeploymentDeadlineParityTests' stub replies with a real zero-failure summary (the artifact
aliases now derive their exit code from failureCount), and register row 127 reflects the merge
and the two new replicated site tables.
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.
Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
I2: topology guide + Component-SiteRuntime + register row 127 state that a site pair using
deployed secrets must run LocalDb replication. I3: row 127 + topology guide state the upgrade
impact (whole artifact deploy fails and reads Out of date until referenced secrets exist
centrally). M6: audit row carries secrets/missing, no removed field; removal-only deploys write
no central row. M2/M3/I1 as-built notes; CLAUDE.md secrets bullet corrected.
The site applies DeployArtifactsCommand.Secrets as the full set (rows absent from it are
soft-removed and DeployedSecretsMirror deletes them on both nodes), while null means
'central shipped no secrets' and touches nothing. The command builder shipped null for a
site with no references, so dropping the LAST secret: reference never removed the secret
from either site node. Found by the Task 13 live gate (check 5: site-a-a and site-a-b
both still held sb-gate-mxgw r1, deployed_secrets.is_removed=0 after the removal deploy).
Central now always ships a list. Old sites ignore proto field 9; the artifact hash is
unchanged (empty and null hash identically); SecretsDeployed stays null and no
DeploySiteSecrets audit row is written when nothing shipped. Regression test
NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes (red
before, green after). Spec notes in Component-DeploymentManager/SiteRuntime + the design
doc's as-built section.
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
vd03 (DEV) upgraded 2b9a7c3c -> fe89363b via the deployment-only winrm channel: COPY_ONLY backup,
55->57 delta script (AddAlarmMesReportable, AddSiteArtifactRevision), ~53 s window, snapshots
identical to baseline, artifacts deployed. Register row 126 now covers the QA pair only.
docker/seed-mes-test-machine.sh (idempotent) builds a standalone TestMesMachine base template -
Left/Right TestMESReceiver + TestDelmiaReceiver (prod attribute names, STATIC), computed alarms
with IsMesReportable, CvdReactor-compatible MesMoveIn/MesMoveOut/ProcessRecipeDownload/AlarmStatus
scripts ported to SetAsync+WaitAsync, and Galaxy-side simulators with reject/no-response knobs -
plus instances SBTEST01/02 and the SimpleAlarmStatusRequest (was a stub) / AlarmStatusRequest
inbound methods in the legacy /mes/(simple)alarmstatus shapes, resolving machines through BTDB.
BTDB rows SBTEST01=990001 / SBTEST02=990002 added on vd03.
docker/verify-mes-test-machine.sh drives the real inbound methods via flow-debug: PASS 25/25.
Joseph Doherty