LogContext is an AsyncLocal and does not cross an Akka message hop, so the run's id now rides the
message: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) on
SetStaticAttributeCommand, AttributeValueChanged (not for an inferred writer), ScriptCallRequest and
the four RouteTo*Requests, plus a new additive SourceExecutionId on GetAttributeRequest,
WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest and WriteTagBatchRequest.
ScriptRuntimeContext stamps its id on the requests it sends, the Instance Actor hands it on to the
DCL, and the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in
AroundReceive (ExecutionLogContext.PushFor: one type test when the message names no execution). The
inbound executor pushes the request's ExecutionId for the method execution.
Tests (TestKit): Instance Actor line carries the message's id, no leak from the sender's ambient
context, scope ends with the message, per-write ids inside a batched SetAttributes, ids forwarded on
DCL writes, ScriptRuntimeContext stamps; DCL adapter lines on single/batch writes carry the id;
inbound executor lines carry the request id; Commons mapping + no serialized member. Each mutation-
proved. Still open on #56: a queryable per-execution sink and a Flow Debugger section. The DCL
actors' own Akka _log lines bypass Serilog (filed #120).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
When the run reset the response flag via SetAttribute before the six-argument
handshake, wait for that reset to echo (shared timeout) before writing batch +
trigger, so the previous run's cached CompleteFlag can't satisfy it. Covers the
unported DB-owned MES/receiver scripts. Pending reset consumed per handshake.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
OtOpcUa's out-of-process script runner publishes 0x80040000 with a null value.
- OpcUaDataConnection: logs entry/recovery once per tag, names the symbolic
status in read errors; never a disconnect (not in the reconnect status set).
- InstanceActor: a Bad update with no value is a quality-only transition
(keep value, quality Bad, stream publish, no script/alarm/waiter re-eval),
matching the connection-drop rule. Bad with a value is unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
TemplateNativeAlarmSource.IsMesReportable + tri-state InstanceNativeAlarmSourceOverride
.IsMesReportableOverride flatten onto ResolvedNativeAlarmSource; NativeAlarmActor stamps every
mirrored condition and InstanceActor the quiet-binding placeholder, so ScriptAlarm,
Alarms.MesReportableAsync() and the #60 wire fields carry it. Additive to the 900-999 severity band.
Migration AddNativeAlarmSourceMesReportable. Management commands, CLI --mes-reportable, Central UI
(template dialog checkbox, instance override select), Transport DTOs/importer/diff and the revision
hash (true-only, so existing hashes are unchanged) carry it.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The site script and alarm on-trigger launchers push the run identity onto LogContext for the
run's lifetime and the host now enables Enrich.FromLogContext(), so log lines written on the
run's own async call chain carry the id. Bounded part only: LogContext does not cross an Akka
message hop, so Instance Actor/DCL/S&F lines and a site_events execution-id column remain open
(documented in the step-logs design doc section 9).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds four instruments on the already-allow-listed ZB.MOM.WW.ScadaBridge meter:
scadabridge.deployments.failed{reason}, scadabridge.store_and_forward.activity{action,category},
scadabridge.site.script.runs{outcome} + scadabridge.site.script.duration{outcome} (ms),
scadabridge.site.data_connections.transitions{state}. Queue depth, DCL connection state and
deploy-applied already existed. No new Meter, so ZbTelemetryOptions.Meters needs no change.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Interface now lives in Commons.Interfaces.Services; AuditLog no longer gets it
transitively through Communication. Implementation stays in HealthMonitoring.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ScriptCallResult and RouteToCallResponse gain trailing Truncated/LogsTruncated,
filled from the site FlowTraceCollector after Complete(); RouteToCallResponseDto
fields 8/9 carry them (sitecommand regen). FlowDebugRunner ORs them into
FlowDebugRunResult on the script entry, and on the method entry through a new
optional truncation sink on RouteHelper.WithTraceSink (carried through every
With* clone). Previously a site run whose collector dropped rows came back as a
silently short list. Older sites send false.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Central's Debug View live stream and Alarm Summary cache always saw false
because neither wire message had the field. Additive proto fields, vendored
C# regenerated with docker/regen-proto.sh (sitestream + sitecommand), mapped
both ways in StreamRelayActor/SiteStreamGrpcClient and SiteCommandDtoMapper.
The live stream publishes Alarm Actor events, which never carried the flag,
so InstanceActor.HandleAlarmStateChanged now stamps it before retaining and
publishing. A pre-#60 site sends the proto3 default. No UI badge yet.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AccessorMirrorPairs rows now carry an ignoredRuntimeMethods list. An entry
must exist on the runtime accessor and must be ABSENT from the mirror, so a
stale or over-broad exclusion fails the theory. The (ScriptLogger, CompileLog)
pair joins the guarded set with FlushDropped excluded (run-end flush called
by ScriptRunLauncher, not script-reachable).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SiteScriptCompileCache now records each eviction sweep on scadabridge.site.script_compile_cache.evictions
(ScadaBridgeTelemetry). It rides the existing ZB.MOM.WW.ScadaBridge meter, which is already in
SiteServiceRegistration.ObservedMeters, so no new allowlist entry is needed; a Host test pins that
the meter stays allowlisted.
Deploys are compile-warmed off the actor thread and reach the four-slot
predecessor-stop queue in warm-completion order (cross-instance ordering is
not guaranteed by design). Since SR-P5 moved the config deserialization into
that warm task, the fifth-sent deploy often finishes warming before one of the
first four, takes a slot, and Queued4 terminated inside the 100 ms no-message
window (16/20 failures under parallel load). The test now waits for the four
in-flight predecessors to terminate, cancels the one still alive, and also
asserts the cancelled redeploy is answered Failed/superseded.
L1: an artifact apply holding the slot past 10 min (ctor-overridable) logs an
Error site event naming it without releasing the slot; PostStop answers
queued applies Failed. L2: a boot-batch instance whose actor cannot be created
emits an Error site event and leaves the deployed set. L3: a failed enable
reverts the committed is_enabled (generation-bumped both ends). L4: the routed
SetAttributes batch Ask is the 30 s per-write bound plus 10 s mailbox slack.
ConnectionActorNames.Sanitize (shared by the DCL manager and the site deploy
gate) keeps ASCII letters/digits and -_.*+:@&=,!~';() only. The manager fails a
connection whose sanitised name is invalid or already used (Error + site
event) instead of restarting; DeployCompileValidator rejects a config whose
connection names collide after sanitising.
HandleOverridesLoaded now sends children a site-local AttributeValueSeed
(stream still gets the AttributeValueChanged). ScriptActor updates its
snapshot, Conditional truth value and Expression last result (seed-only eval,
never fires); AlarmActor evaluates it as a normal change.
DeployCompileValidator.WarmCompileCache dedupes script bodies/trigger
expressions across a batch and compiles them with Parallel.ForEach (half the
cores). CreateInstanceActorBatch starts the next batch's warm as it arms the
stagger timer; creation order and delay are unchanged. The warm's parsed
configs are reused at creation.
The deploy compile warm parses once off-thread; the parsed FlattenedConfiguration
rides DeployCompileWarmed to the gate (new Validate(FlattenedConfiguration)
overload), the redeploy buffer, CreateInstanceActor's DCL ensure and a new
InstanceActor constructor overload. Stored-row paths parse once inside
CreateInstanceActor; an unparseable stored row still fails init via S6.
Adds SiteStorageService.GetAllDeployedConfigMetadataAsync. The sweep decides
from the projection, then fetches single rows for its starts/restarts off-
thread and re-validates them (generation guard, revision match, pipeline/
running guards), releasing the per-revision claim of any repair that does
not act. Boot load unchanged.
One Ask per attribute became one site-local InstanceGet/SetAttributesRequest
answered in a single pass, keeping the per-attribute GetAttributeResponse /
SetStaticAttributeResponse outcomes. The single-write path is refactored to a
Task<SetStaticAttributeResponse> outcome shared by both paths; static writes
and rejections still reply synchronously.
Each apply upserts its full set then deletes everything else, off the actor
thread; two overlapping commands interleaved and deleted artifacts central
never removed. The Deployment Manager now runs one apply at a time, queuing
the rest in arrival order; the slot is released by a self-message that lands
behind the apply's own DCL push.
SR-U1: children are created before the async override load lands, so their
trigger snapshots kept the deploy default forever after a restart/failover/
enable/sweep start. HandleOverridesLoaded now republishes every changed value
through PublishAndNotifyChildren(evaluateWaiters:false), and skips attributes
a static write touched before the load (newer than the stored row).
SR-M3: PostStop cancels each WaitForAttribute timeout and replies
'instance stopped' instead of leaving scripts to time out and timeouts to
dead-letter.
An instance unique name Akka refuses as a path element (space, '#', '/',
non-ASCII) made Context.ActorOf throw on the singleton's own thread; the
default Restart stopped every Instance Actor on the site. DeployCompileValidator
now checks the instance + script/alarm/native-source actor names with
ActorPath.IsValidPathElement (the Commons ActorNameRules mirror is stricter
than Akka for " ( ), pinned by a parity sweep), CreateInstanceActor catches
InvalidActorNameException and every caller reports it, and the Instance Actor
skips a refused child name like a compile failure.
Comment/XML-doc changes only:
- add summaries to ~10k test methods, fixtures, fakes and helpers
(+ <param> on theories, <returns> on async Task methods,
<inheritdoc/> on interface implementations the checker can resolve)
- strip ~2.6k project-tracking references (WP*, P*-*, row-NN, Task N,
#NN, Component-0NN) from comments, keeping the explanatory text;
reword false positives in comments only
- real docs on fakes implementing BCL/library interfaces
(IDisposable, ILogger, IOptionsMonitor, IAsyncLifetime)
- reword the last 5 src/ false positives (CWE-1236, failures-1,
year-0001, M365)
Only non-comment edits: four one-line nested classes split across lines
so members can carry docs. Build 0 errors / 0 warnings.
Rig finding on the upgrade restart: both nodes' rows were older than the stale threshold
(the old build never refreshed them), so node-a's reporter pruned node-b's row while node-b
wrote a fresh one, and the replicated delete won. The reporter now prunes nothing during its
first refresh interval after start-up, and each mirror pass re-writes its own row if it no
longer holds what the mirror last wrote.
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval
(10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and
deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the
refresh). The delete is conditional on the checked_at it read, so a node that came back
keeps its fresh row; a returning node re-writes its row on its first pass.
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.
Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
Final review M2: HandleDeployArtifacts now sends ApplyArtifactDataConnectionsToDcl after the
shipped secrets are written and mirrored (from a finally, so the push still goes out if the
secrets step fails), so a connection created by the same deploy resolves a newly shipped
secret on first connect. Definitions are still stored at the same point; response semantics
unchanged.
Final review I1: a name already ledgered at the shipped revision is now read back every
pass; a missing, tombstoned or edited local row is healed through the mirror-owned re-stamp
path (bump, apply, read back, invalidate) before verify, instead of a local delete being
misreported as a KEK mismatch that blocked every later deploy. Drift logs Warning once per
occurrence. M4: row_json that cannot be parsed becomes a Failed entry for that name via the
new SiteStorageService.GetDeployedSecretsTolerantAsync; an unreadable removed row is still
honoured by name. M10: invalidator ctor doc corrected.
Fix round 2 (review task-9-rereview-1.md), both Minors:
1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
the local row happens to be a tombstone. A live local row this node already owns but whose
ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
message is now only ever produced for a name that is NOT in the ledger. New test:
Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
re-stamp followed by a genuinely new central revision converges and re-invalidates the
cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
AEAD-bound" property against the real cipher rather than by source inspection alone.
Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
ZB.MOM.WW.Secrets.Abstractions).
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
Fix round 1 (review task-9-review.md), all rulings:
1. (Important 1) ApplyReplicatedAsync is no longer trusted blindly. Every apply is
confirmed by a store.GetAsync read-back before ledgering: a tombstoned local row
(routine remove-then-re-add) is safely re-stamped past the tombstone and re-applied
(confirmed neither UpdatedUtc nor Revision is AEAD-bound in AesGcmEnvelopeCipher, so
the bump cannot break decryption); a live local row with different ciphertext (a
strictly-newer out-of-band secret) is left untouched, never ledgered, never verified
against, and reported with a distinct Failed message so a later central tombstone can
never delete it via this mirror.
2. (Important 2) Per-row try/catch (rethrow only on ct.IsCancellationRequested) so one
poisoned row (e.g. an unrecognised ContentType from a newer central) no longer blocks
every row after it.
3. (Important 3) Optional ISecretCacheInvalidator, invalidated after every successful
apply/delete and before verify, so a rotated/removed secret's plaintext cannot be
served stale for the remainder of the resolver's TTL.
4. (Minor 4) Loop and VerifyAsync OperationCanceledException handling now filters on
ct/stoppingToken.IsCancellationRequested specifically, rather than swallowing every OCE.
5. (Minor 5) Class remarks note the mirror assumes a node-local ISecretStore; Task 11
will reject SqlServer secrets replication on the Site role.
6. (Minor 6/7) Added cancellation + concurrent-callers tests, dropped the redundant
FailFor (tests now use ThrowFor/ReturnNullFor to exercise distinct decrypt-failure
paths), and Dispose() now disposes the gate SemaphoreSlim.
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (24/24); full SiteRuntime.Tests suite 876/876.
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
A null resetValue encoded to null, which wrote an empty string and then handed the
waiter a null target (= any change), fast-pathing on the stale cache and reinstating
the race the overload exists to close. The context parameter is now non-nullable and
the accessor throws ArgumentException before any write. Also: additive-timeout note on
all surfaces, batch-failure message no longer names the bool overload, and the
error-message/batch-failure branches are pinned.
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
Joseph Doherty