TimeoutSeconds is the per-system HTTP timeout ExternalSystemClient reads at
call time and is deployed to sites (proto field 6), but Transport bundles
silently dropped it: the DTO never carried it, export never wrote it, the
importer's Add path reset it to 0, the Overwrite path left the target's
stale value untouched, and ArtifactDiff never compared it (so a bundle
whose only change was a tuned timeout previewed as Identical).
Adds TimeoutSeconds as an additive-optional trailing field on
ExternalSystemDto (schemaVersion stays 1.1, bundleFormatVersion unchanged)
and threads it through EntitySerializer (export + import), BundleImporter
(Add and Overwrite), and ArtifactDiff.CompareExternalSystem. Old-bundle
semantics pinned: an absent field (null) falls back to the entity default
on Add and leaves the target's existing value untouched on Overwrite/diff.
Beefed up the RoundTripEquivalenceTests external-system fixture with a
non-default TimeoutSeconds so the reflection round-trip guard actually
exercises the field instead of comparing default against default.
Swept the rest of the ExternalSystemDto shape against
ExternalSystemDefinition field-by-field; no other operative field is
missing.
Row 65 — trigger FIRED, fixed. BundleUnlockAttempt rows were pruned only by
nodes actively serving unlocks, so a cluster that stopped importing entirely
left its last rows in place forever. The row's own trigger was "Transport gains
a central scheduler a prune tick could ride" — it has one now:
BundleSessionEvictionService, the central-only 1-minute session sweep added by
the transport-session fixes. It now also ticks the new
IBundleUnlockAttemptWindow.PruneAsync. Both drivers share the ONE 15-minute CAS
cadence claim, so ticking on a 1-minute sweep costs at most one bounded DELETE
per quarter hour and the two can never prune concurrently. PruneAsync is
contractually non-throwing (unlike TryRegisterAttemptAsync, which throws and
lets the limiter own the fail-open decision) — a background sweep has no caller
to report to and must survive a database blip. Both Transport docs updated;
three tests in FleetWideUnlockThrottleTests, incl. the retention sweep running
with no unlock traffic anywhere in the test, which is the whole point.
Row 58 — closed with a pin, and the OLD pin was found blind. The existing
Deployments_Dispose_UnsubscribesFromNotifier is behavioural: it is satisfied by
the _disposed guard alone, so deleting the `-=` from Dispose leaves it green
while the singleton notifier keeps a delegate pinning the circuit-scoped
component. Verified by mutation — with the `-=` removed the old test still
passed and only the two new pins failed. Added:
* a structural retention pin asserting the notifier's invocation list actually
shrinks to 0 on Dispose;
* a source scan pinning the row's stated TRIGGER ("a second page subscribes,
or any subscriber lands without a matching Dispose unsubscribe") — every
CentralUI file that subscribes must also unsubscribe and be IDisposable,
with a non-empty guard so the scan cannot pass vacuously.
Row 60 — closed with a pin, and the register's premise corrected.
GlobalArtifactSnapshot does NOT hold EF-materialised entities: it is projected
into pure Commons DTOs before the fan-out. The EF state that genuinely crosses
into the parallel send is the tracked Site entities the lambda closes over, and
that is safe only while lazy loading is off. Pinned in ConfigurationDatabase
.Tests two ways — no proxies extension on the built DbContextOptions, and no
EntityFrameworkCore.Proxies package reference anywhere in the repo (the
structurally stronger half: UseLazyLoadingProxies() is not a compilable call
without it). The real invariant is now spelled out at the fan-out itself.
Row 59 — no product change; added the missing tripwire remark stating the
constraint the safety argument rests on (serial by design, one scoped DbContext,
do not Task.WhenAll; if it gets slow the fix is a scope PER SITE, not a shared
scope under fan-out). Line numbers in the register have drifted to :528-545.
Rows 36, 37, 64: verification only, no change — see the sweep report.
BundleSessionObservabilityTests' two gauge facts asserted absolute values
(Assert.Equal(0, baseline), Assert.Equal(2, ReadGauge())) against what is a
single process-global provider slot — the same latent shape as the row-32
QueueDepthGaugeTests flake, correct only while this file is the slot's sole
registrant. Converted to the delta-over-scoped-action pattern the row-66
rejection-counter facts in the same file already use: a new ReadGaugeDelta
helper measures how far the instrument MOVED across an action driven by this
test's own store, so an inherited non-zero baseline cancels out.
Two assertions got stronger in the process. The post-clear check was
Assert.Equal(0, ReadGauge()) — "the instrument reports zero", which any
unregistered slot satisfies; it is now "opening more sessions on the detached
store moves the gauge by nothing", which is the actual claim. And the
compare-and-clear fact gained a negative control: the superseded store's opens
must NOT reach the instrument, so a slot still pointing at oldStore can no
longer satisfy the assertion by coincidence when both stores hold equal counts.
Fail-before (temporary probe, not committed): with a foreign provider returning
7 registered before the service starts, the OLD body fails
`Assert.Equal(0, baseline)` with `Expected: 0 / Actual: 7`; the NEW delta body
passes the identical scenario.
Honest residual documented on the helper: a gauge has one slot and
last-writer-wins, so a registrant that clobbers it BETWEEN the two reads is
unrecoverable read-side — that needs row-32's mutual exclusion. It cannot arise
today (this is the only file in the assembly constructing a
BundleSessionEvictionService, and test assemblies are separate processes); a
second such file should bring the collection treatment with it.
Transport.Tests: 180/180 green.
An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.
The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.
Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
(0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
the existing TimeoutSeconds "omit to preserve" convention. Both additive.
Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.
Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
sitecommand; csproj Protobuf item re-commented, no message types added or removed.
Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
appended so existing ones are untouched), apply path, and the legacy migrator's
explicit column list all carry them.
Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.
Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.
No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.
Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).
Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
Row 53 shipped the scadabridge.central.transport.import_sessions.open gauge
and a Warning log on a cap hit, but a log line is not something an operator
can alert on. Adds the companion counter
scadabridge.central.transport.import_sessions.rejected
on the existing ZB.MOM.WW.ScadaBridge meter, so no ObservedMeters allowlist
change is needed (an unlisted meter exports nothing, silently).
The emit point is deliberately narrow. BundleSessionStore.Open tests the cap
TWICE — once on entry and again after EvictExpired() — and only the second
test is a rejection: a store full of merely expired sessions reclaims a slot
and admits the caller. Counting the first would make the series fire
routinely on any node whose sweep had not yet run, turning a page-worthy
signal into background noise. Re-Opening an already-open session id skips
the cap block entirely and is likewise not counted.
The two instruments are diagnostic as a pair: rejections rising while the
gauge oscillates below the cap is genuine concurrency (raise the cap);
rejections rising while the gauge sits pinned at the cap is a session leak
(register row 43's signature), where raising the cap only defers the problem.
Tests assert a delta over a scoped action rather than an absolute value, so
they do not inherit the register row 32 static-state flake that makes
QueueDepthGaugeTests fail only under the full concurrent suite. Negative
controls cover under-cap, expired-reclaim, and duplicate-session-id.
Tests: 11 passed in BundleSessionObservabilityTests (6 existing + 5 new).
Nothing counted open import sessions and nothing logged a rejection at
MaxConcurrentImportSessions (default 8), so the register-row-43 session leak
could only be inferred from rotating test failures.
- Gauge scadabridge.central.transport.import_sessions.open on the existing
ZB.MOM.WW.ScadaBridge meter — already in SiteServiceRegistration.ObservedMeters
(the ZbTelemetryOptions.Meters allowlist), so no telemetry config change is
needed and the series is not silently dropped. Provider-backed, mirroring the
StoreAndForward queue-depth gauge: registered by BundleSessionEvictionService
.StartAsync (Transport's only lifecycle seam) and identity-checked-cleared on
StopAsync so a late stop cannot clobber a newer registrant.
- IBundleSessionStore.OpenSessionCount counts only UNEXPIRED sessions. The raw
dictionary count would report the node as pinned at the cap for up to a
one-minute sweep interval after a burst had drained — the exact false positive
the instrument exists to rule out.
- BundleSessionStore.Open logs Warning with open count, cap and TTL when it
rejects. Fires only on a real cap hit: a store full of expired sessions is
reclaimed first and logs nothing.
- Docs: docs/components/Transport.md + docs/requirements/Component-Transport.md.
Tests: Transport 173 passed (+6 new — Warning content, two negative controls,
unexpired-only counting, gauge register/clear, no-clobber-on-late-stop).
BundleUnlockRateLimiter was registered (ServiceCollectionExtensions), injected
into BundleImporter, unit-tested, documented on
TransportOptions.MaxUnlockAttemptsPerIpPerHour = 10 and pinned in
appsettings.Central.json — but _unlockRateLimiter was never invoked. A
brute-force passphrase attacker against a stolen bundle was bounded only by the
per-bundle 3-strike lockout, which a re-export side-steps (new ContentHash).
LoadAsync now registers every unlock attempt against the limiter. Ordering is
load-bearing: AFTER the per-bundle lockout check (hammering a locked bundle
spends no hourly budget) and BEFORE the decrypt (so a denial burns no per-bundle
strike — otherwise the throttle would be a way to lock a bundle someone is
legitimately importing). Over budget throws the new
BundleUnlockThrottledException, fail-closed with a clear message on both
surfaces; it lives in Commons because the management surface (Commons-only
references) must catch it to return a curated error rather than MapFault's
generic internal-error text.
Client key plumbing:
- Central UI: new scoped CircuitClientAddressProvider snapshots
Connection.RemoteIpAddress at circuit establishment — the same constraint and
the same pattern as CookieAuthenticationStateProvider, since HttpContext is
null by the time a circuit submits a passphrase.
- CLI/management: additive ManagementEnvelope.ClientIp, stamped by
ManagementEndpoints and threaded through DispatchCommand to the preview/import
bundle handlers.
- No attributable address => a single SHARED "<unattributed>" bucket, not a
bypass: unknown clients get a stricter budget, never none. Per-client
isolation is only real behind a ForwardedHeaders-trusted proxy, the same
caveat the login throttle carries (Component-Security.md).
Docs: Component-Transport.md + docs/components/Transport.md record the two-level
ordering, the client-key sourcing, the shared-bucket fallback and a
troubleshooting entry for the new error.
Tests: 6 importer tests (cap enforced; denial != passphrase failure, no strike
burned; per-client isolation; shared unattributed bucket; unencrypted bundles
consume no budget; a locked bundle spends no hourly budget) — 4 of the 6 fail
with the limiter call disabled, i.e. against the dead-code state. Plus
ManagementActor tests pinning the ClientIp forwarding and the curated
throttle error.
Only BundleImporter.ApplyAsync released an import session, so every path that
opened one and did NOT apply held one of MaxConcurrentImportSessions (8) — and
the DECRYPTED bundle content that T-007 deliberately zeroes on the apply paths —
for the full BundleSessionTtlMinutes (30). Eight abandonments locked out all
imports on the active node; observed live (R6: 3 back-to-back suite runs held 6
of 8 slots until central-b was restarted).
New IBundleImporter.Abandon(sessionId): the same zero-then-remove release
ApplyAsync performs, idempotent (double release / unknown id / expired id are
no-ops) and only ever given the random-GUID id its caller opened, so it can
never evict another circuit's session.
Wired into every abandonment path:
- Central UI wizard — ReleaseSession() from new upload, TryLoadAsync (before it
overwrites _session), Back-to-Upload / ResetSessionState, the 3-strike lockout
reset, and Dispose (navigation away / circuit teardown).
- CLI transport preview — a preview never applies, so nothing released it at
all; now released in a finally.
- CLI transport import — covers the pre-apply aborts (blocker rows, unresolved
site/connection mappings).
Class doc's "an abandoned wizard does not leak" claim is true again (it only
ever covered the staged plaintext temp file). Component-Transport.md +
docs/components/Transport.md now record the explicit-release contract and that
BundleSessionEvictionService is a TTL backstop, not an abandonment path.
Tests: 4 bUnit tests over a REAL BundleSessionStore (dispose / back-to-upload /
apply-then-dispose idempotency / concurrent-circuit isolation) — 3 of the 4 fail
against the pre-fix page; 4 importer tests (release + zeroing, idempotency,
cross-session isolation, and the slot-cap regression).
#05-T24. Three low-severity cleanups, each TDD'd:
- TransportOptions.MaxConcurrentImportSessions (8): BundleSessionStore.Open rejects
a new session past the cap (evicting expired entries first), bounding the N×~200 MB
decrypted-content footprint. Soft cap (count/add not atomic).
- ScriptCompiler.TryCompile now joins ALL forbidden-API violations / compile errors
(string.Join instead of [0]) so an operator sees every problem at once; the T15
verdict cache stores the joined string.
- SemanticValidator emits an advisory warning when a CallScript target resolves
ONLY via the composed leaf-name fallback (dynamic child path, not statically
verified) instead of silently accepting it.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
#05-T23. ArtifactDiff.CompareTemplate gains optional folderNameById /
templateNameById maps; FolderNameOf / BaseTemplateNameOf / CompositionTargetNameOf
resolve the existing template's FolderId / ParentTemplateId / composition
ComposedTemplateId to real names (falling back to the <id:N> placeholder only
when a map is absent or misses, keeping existing unit-test callers valid).
PreviewAsync builds templateNameById from GetAllTemplatesAsync (parent/composition
targets can be any template, not just bundle-named ones) and threads both maps
through. Fixes spurious Modified on every re-import of a foldered/derived/composed
template. End-to-end preview test + two ArtifactDiff unit tests.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
New TemplateChildEquality is the one place "has this template child changed?" is
decided, comparing the COMPLETE writable field set per entity (attributes now incl.
ElementDataType + LockedInDerived; alarms incl. the on-trigger script by name +
LockedInDerived; scripts incl. MinTimeBetweenRuns/ExecutionTimeoutSeconds +
LockedInDerived; native sources per T5). ArtifactDiff.CompareTemplate and all four
BundleImporter.SyncTemplate*Async predicates now route through it — closing the
class of bug where Preview reported Identical for an artifact an Overwrite would
mutate. The attribute sync passes a value-normalised DTO so List-value re-imports
stay idempotent; the alarm on-trigger change now surfaces in both diff and sync.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Follow-up from the T5 code review: assert an idempotent Overwrite emits zero
TemplateNativeAlarmSource* audit rows, and round-trip IsInherited/LockedInDerived
as true through both the serializer and the importer BuildTemplate path.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
New TemplateNativeAlarmSourceDto + init-only TemplateDto.NativeAlarmSources (empty
default, additive — IsInherited placeholders carried for the collision detector).
Exported in ToBundleContent, consumed in FromBundleContent + BuildTemplate, and a
new SyncTemplateNativeAlarmSourcesAsync runs the Overwrite add/update/delete child
sync with per-change audit rows. ArtifactDiff.CompareTemplate now DiffChildren over
native sources so Preview reports a NativeAlarmSources field change. Proven by an
end-to-end flatten test: an instance native-alarm-source override that dangled
pre-fix now resolves against the re-imported template source.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Adds a trailing optional `bool LockedInDerived = false` to TemplateAttributeDto,
TemplateAlarmDto and TemplateScriptDto (mirrors the ExecutionTimeoutSeconds
additive precedent — no schemaVersion bump). Populated at export
(ToBundleContent), consumed on import (FromBundleContent, BuildTemplate, and all
three SyncTemplate*Async changed-predicate + copy + new-entity paths, incl. audit
payloads). Old-form bundles that lack the property still deserialize to false.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Add DataType? ElementDataType to TemplateAttributeDto (optional, default null
for backward-compat with old bundles). Map it in both directions in
EntitySerializer (export + FromBundleContent) and in all three
TemplateAttribute construction sites in BundleImporter (BuildTemplate,
SyncTemplateAttributesAsync add-path, and SyncTemplateAttributesAsync
update-path including change-detection). Two new round-trip tests in
EntitySerializerTests confirm List attributes survive export→import and that
old DTOs with null ElementDataType import cleanly.
Renames the 13 SCADALINK_* runtime env vars → SCADABRIDGE_*, the ScadaLink__
.NET config keys → ScadaBridge__, the stale ScadaLink.Host.exe assembly name
→ ZB.MOM.WW.ScadaBridge.Host.exe, the scadalink_app SQL login → scadabridge_app,
and residual identifiers/comments/docs. Migration records (prior rename
tooling/design, DB-rename helper, this scrub script) carved out.
Adds tools/scrub-scadalink-refs.sh.
Joseph Doherty