VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and
RoslynVirtualTagEvaluator blocked on the run guard with
.GetAwaiter().GetResult(), so every script run parked a dispatcher thread for
up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0
refactor for virtual tags.
- IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the
snapshot overload, so the null evaluator and fakes are unchanged).
- RoslynVirtualTagEvaluator implements it without blocking: passthrough and
compile failures complete synchronously, a first-use compile goes to the
pool, the run is awaited. The blocking overload now delegates to it.
- VirtualTagActor: a completed task is applied in the same turn, as before;
an incomplete one switches to an "evaluating" behaviour that stashes every
dependency/reassert message until a private completion message arrives,
then applies the result and unstashes. Each trigger is still evaluated
against its own inputs, results are applied/published in trigger order and
the dedup/Bad-transition state sees the same sequence. The dependency map is
not touched while a run reads it.
Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a
thread, and running independent scripts in parallel is not semantics-neutral
(pool consumption bound, cross-tag publish order, timer/pump exclusion for
Calc; state-machine/persist boundary, transition timestamps and the
store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750
design's P0 entry.
Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow
script no longer starves a sibling actor and the slow actor answers Identify
(fails when the actor calls the blocking Evaluate); triggers during a run are
evaluated afterwards in order against their own inputs; a reassert during a
run is applied after it; a faulted evaluation takes the evaluator-threw path
(the ordering/reassert tests fail with the stash removed).
RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return
an incomplete task (fail without the override); cheap outcomes and timeouts
match the blocking overload.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.
- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
builds the engine through CreateScriptedAlarmEngine, which passes it
explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.
Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.
GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.
Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.
Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.
Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.
DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
token cancel = Timeout
abandon = Timeout + min(Timeout/2, 2 s) (RunBound)
queue budget = RunBound
child replies <= queue budget + RunBound (ReplyBound)
host Ask = ReplyBound + 1 s
gateway Ask = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.
- DriverHostActor takes writeBounds + writeClock, spawns children with them,
and runs its Ask deadline as a token on the write clock (Infinite Akka
timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
host): the queue budget counts mailbox time too, and a write past it is
failed BadTimeout and never started - including on arrival. Stale queued
writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
style as ScriptingOptionsBinding): unparsable / non-positive -> default,
above 5 min -> clamped, and each tier whose default Write timeout exceeds
the write timeout is named in a Warning (its policy would never take
effect). The derived bounds are logged at Information. Per-instance
ResilienceConfig Write overrides arrive with a deployment, so
DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.
Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
an 8.5 s step and succeeds; a token-ignoring write is answered by the
child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
the queue budget is failed and never reaches the driver; positive control
at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
caught at startup with Tier B and C named (not A); bad values fall back
with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.
Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.
ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
HistorianBackendRegistryTests (#682) still expected the Gateway data source
to advertise the two AVEVA-native aggregates. Lane H2 (R08:2.1) refuses
them by default until historiangw#27, and the lane branch never updated
this Host test, so it failed after the rebase. The default case now expects
the ten raw-evaluated aggregates, and a new case pins that
ServerHistorian:Gateway:EnableNativeTimeWeighted=true, resolved through the
Host's registry, brings TimeAverage and Total back.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The #733 miss budget (MissBurst / MissRefillPerSecond) was process-wide, because
the impersonation seam did not carry the client's address, so one noisy client
could drain it for everybody while the known-id set was unavailable.
- IOpcUaUserAuthenticator gains AuthenticateUserNameAsync(username, password,
IPEndPoint? remoteEndpoint, ct) (default: ignore it and delegate); the server
calls it. IApiKeyStore gains the matching FindAsync(..., IPAddress?, ...).
- The SDK (1.5.378) exposes no client address publicly. SecureChannelRemoteEndpoint
reads it from the TCP listener's private channel map by reflection (through the
protected TransportListeners, exposed by OtOpcUaSdkServer), keyed by the
session's secure channel id. Fail-soft: null => process-wide accounting only.
- BoundedApiKeyStore charges a miss to the client's own bucket (keyed by IP
address) and then to the process-wide one, which stays as the backstop. A
refusal by either takes from neither; a hit refunds both. New options
Security:ApiKeys:MissBurstPerEndpoint (10) / MissRefillPerSecondPerEndpoint (2),
0 disables; telemetry outcome endpoint_miss_budget. At most 4096 addresses are
tracked; idle buckets are pruned and an address beyond the cap is charged
process-wide only.
Tests: BoundedApiKeyStorePerEndpointTests (noisy client vs quiet one, process-wide
backstop, refund + refill, off switch, authenticator forwards the address) and
over the wire ApiKeyMissBudgetPerEndpointTests (activation hands the authenticator
the loopback endpoint; real authenticator + bounded store refuse a client's 3rd
and 4th unknown key without a store read). Each new behaviour test fails with its
fix mutated out. Host.Tests 224/0, OpcUaServer.Tests 1019/1 (the one failure is the
wall-clock NodeManagerLockTimeBenchmarkTests under suite load; passes alone).
Docs: security.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AbLegacyDriver stamped LastSuccessfulRead (init, read, write), the read's sample
timestamps and host-state change times from raw DateTime.UtcNow; only the backoff
path read the injectable Clock. All of them now read Clock(), so
NativePcccBridgeStatusHealthTests drives a manual clock (init at T0, read at T1)
and asserts LastSuccessfulRead == T1 exactly instead of spin-waiting for the wall
clock to pass the init stamp (#767). Without the driver change the 6 tests fail
(init stamp is wall-clock, not T0).
Directory.Build.props sets TreatWarningsAsErrors for every project since #766, so
the 75 per-csproj copies (64 src + 11 tests) are removed and the props comment
says it is now the only place. tools/mem-probe keeps its explicit opt-out.
Release build: 0 warnings, 0 errors; a planted CS0168 still fails the build in a
src project and in a test project that used to carry its own copy.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Review m-1. The bind moves from Program.cs into
Host.Configuration.ScriptingOptionsBinding.Bind, which binds the
section and clamps it. Program.cs calls it, and
ScriptingOptionsBindingTests drives it with in-memory configuration.
Every documented key takes a non-default value, so a misspelled
property or key goes red instead of silently falling back to the
defaults. The tests also cover a clamped bind (1e300 / 1024 / 0, three
adjustments), an absent section, and the sibling LogFilePath /
LogTopicMinLevel keys leaving the bind untouched.
Refs #731#734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
BoundedApiKeyStore kept the set (volatile) and its DateTimeOffset stamp (non-volatile, 16 bytes) in separate
fields, so a reader could pair a new set with an old stamp or read a torn stamp; they are now one immutable
snapshot behind a volatile reference. An inner store that cannot list ids now switches the set off with a
per-instance flag instead of writing KnownIdSet = false into the shared ApiKeyStoreGuardOptions.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Coordinator ruling (I6): until ServiceLevel consumes IOpcUaEndpointState (F-C1), the probe must not sit on
the Active tier of a fused node, because Traefik routes the AdminUI on /health/active. Driver-only nodes keep
it on Ready + Active; admin-only nodes do not register it. Pinned per role.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
BoundedApiKeyStore's negative cache was keyed by key id, so a flood of DISTINCT well-formed
otua_<random>_x credentials missed it every time and cost one SQL query per attempt under the
SDK's SessionManager-wide lock, no credential needed (R03:3.3).
- IApiKeyStore.ListKeyIdsAsync (default null = unsupported; ConfigDbApiKeyStore lists every
ApiKeyId): the bounded store keeps a known-id set refreshed every StoreCacheSeconds
(single-flight, failed refresh keeps the previous set) and refuses an unknown id without a store
read ({outcome=unknown_id}).
- A process-wide miss budget (token bucket, Security:ApiKeys:MissBurst=100 /
MissRefillPerSecond=20, ApiKeyStoreGuardOptions) bounds lookups that reach the store and miss
while no set is available; a hit refunds its token. Per-endpoint accounting needs the
OpcUaServer impersonation seam to carry the client address — follow-up.
- docs/security.md: the new knobs, and the known-cluster gate's documented fail-open window at
central boot (R03:4.2) plus the async PSK refresh (R03:3.4).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
(JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
/auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
A failed SDK start (port in use, PKI unwritable, bad OpcUa:* override) was one Error line: the
node stayed /health/ready 200, joined its mesh, could be elected driver Primary and acked deploys
through the Null sink while every client got connection-refused.
- IOpcUaEndpointState / OpcUaEndpointState: the start outcome is a fact the hosted service
maintains (NotStarted / Started / Failed / Stopped + reason).
- OpcUaEndpointHealthCheck ("opcua-endpoint", Ready + Active) is Unhealthy unless Started;
registered by AddOtOpcUaHealth(hasAdmin, hasDriver) on driver-role nodes.
- A driver-only node rethrows the start failure (Program.cs's fatal handler exits non-zero so
the supervisor restarts it); a fused node keeps its AdminUI up but is visibly not ready.
- StopAsync detaches the sinks and then STOPS the SDK (the port closes, clients get
connection-refused rather than a stale, inert endpoint); DisposeAsync is idempotent with it.
- The ServiceLevel calculator / redundancy election live in Cluster + Runtime (not this lane);
the fact is exposed for the Wave 3 follow-up to consume.
Tests: port-in-use ⇒ state Failed, probe Unhealthy, real /health/ready and /health/active 503
(positive control: 200 once Started); driver-only rethrows; StopAsync closes the port and
DisposeAsync twice is a no-op; the probe registration is on both tiers for driver nodes only.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.
Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.
Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
1. The at-or-after bound read on a forward-only backend (Gateway) ends at
now + ReverseRawReader.LookAhead instead of DateTime.MaxValue (the gateway
only checks start <= end, so a year-9999 end reached AVEVA); a bound beyond
the horizon is not found without a read. SqlServer/QuestDB keep MaxValue:
they bind it safely and may hold samples further ahead. The Gateway fake now
refuses far-future ends, so the missing-end-bound test is no longer vacuous.
3. No data and no bound found (whole read empty) => GoodNoData, no values —
the pre-bounds answer (6.5.3.2: success only if one or both bounds exist).
A one-value page whose start bound is missing first checks there is data or
a found end bound.
4. ServerHistorian:HonorReturnBounds (default true); false ignores ReturnBounds
as before #691, including the modified-read Bad_InvalidArgument.
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).
- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.
Co-Authored-By: Claude <noreply@anthropic.com>
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.
Co-Authored-By: Claude <noreply@anthropic.com>
- OtOpcUaNodeManager overrides HistoryUpdateData / HistoryDeleteRawModified / HistoryDeleteAtTime,
block-bridging to the optional IHistorianDataUpdater under the shared HistoryRead batch limiter +
deadline. Insert/Replace/Update map to HistoryUpdateMode; Remove, IsDeleteModified, events and
structure data are BadHistoryOperationUnsupported. Per-item codes stitched back in request order
(MinValue SourceTimestamp => BadInvalidTimestamp, untypeable value => BadTypeMismatch); success
clears the base's pre-seeded Unsupported error.
- HistoryWrite AccessLevel bit OR-ed in for historized nodes when an updater is wired (raw + UNS);
it is dispatch-load-bearing in the SDK base. OtOpcUaSdkServer.SetHistorianDataUpdater; the hosted
service assigns it from an optional HistorianUpdaterHolder before SetSink, clears it on stop.
- AuthorizeHistoryUpdate: ACL on => scoped OpcUaOperation.HistoryUpdate decision (replaces the role
check); ACL off => #563 key re-check + new OpcUaDataPlaneRoles.HistoryUpdate role, conferred by
ApiKeyInterimRoleMapper only from a Cluster-scoped HistoryUpdate grant (WriteOperate does not).
- AdminUI ACL editor offers NodePermissions.HistoryUpdate.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
ConfigureSharedLogger had no production caller, so the pool's only
diagnostics (first-wins options conflicts, scheduler/close failures)
went to NullLogger. DriverFactoryBootstrap now configures it from the
driver registry factory and from the AbCip/AbLegacy probe factories
(admin-only nodes reach the pool only through Test Connect). Wiring tests
reset the pool to a null logger first, so removing the call goes red
(verified).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The old AdminUI cannot author the per-driver transport key (#576), so the
'pin before upgrading' recipe had no working path. A node-level appsetting
(env Drivers__AbCip__DefaultTransport) now supplies the default for AbCip
drivers with no key: explicit key > node default > compiled Native. Bound
and validated unconditionally (bad value fails host start); the driver
factory and the Test Connect probe both read it so they agree. AbCip.md
upgrade recipe rewritten around the key; Configuration.md documents it.
Also documents first-wins for AllowPacking/probe and host-alias throttles
(final review M6) and moves the status-bit note out of AbCip.md (M5).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.
BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.
ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.
An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).
RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.
Host.Tests 137/137.
Gitea #561 ("production writes a 0-byte log") is closed as not-a-defect. The
production log on wonder-app-vd03 listed as 0 bytes with LastWriteTime ==
CreationTime; opening it showed 4,658,893 bytes of correct output, written
continuously since service start.
On Windows the directory entry's size and last-write time are not refreshed while
the writing process holds the file open, so Get-ChildItem, dir and FileInfo.Length
all report size-at-open for the life of the handle. A live, healthy log is
indistinguishable from a dead sink by metadata alone — which is what made the
false reading survive six weeks of investigation.
That same trap was in the 29a202f5 instrumentation: CheckFileSinkGrowth measured
with FileInfo.Length, so it would have reported "self-check FAILED: the sink did
not grow" at every boot on a perfectly healthy Windows service — and a
self-check's false positive reads as confirmation of the very bug it was added to
find. MeasureFamilyBytes now reads Length through an opened FileStream:
FileShare.ReadWrite to open it at all while the sink holds its write handle, and
FileShare.Delete so the probe never blocks a roll or retention delete.
Both instruments are kept. A sink that cannot write its file is still a real and
otherwise-invisible failure — a sibling repo reproduced it via handle contention,
where RollingFileSink logs the lock-out to SelfLog and falls through to _001. Only
the two doc claims that are now false are corrected: the class summary's
"production ran that way for six weeks" and the warning text's "#561 failure mode".
Tests: A_working_file_sink_passes already measured while Serilog held the file
open — precisely the broken case — and passed throughout, because it only ever ran
on macOS, where the length is honest. That platform blindness is now recorded in
the class summary, alongside two tests pinning the sharing contract the fix
depends on.
Four-minor jump 0.2.3 → 0.6.2. The release line exists because a deploy
renaming the app directory took the MxGateway credential store with it
(2026-08-09, two silent days): Secrets:SqlitePath must now be absolute AND
outside the content root, rejected at boot rather than relocated. 0.6.2 over
0.6.1 is message-only (the rejection text now teaches the mounted-volume
case and forecloses the override hunt) — the live rig evidence below was
gathered at 0.6.1 and the validator behavior is identical.
Adoption changes beyond the pins:
- appsettings.json drops the relative "otopcua-secrets.db" (now boot-fatal);
dev gets the library's absolute per-user default, deployments supply their
own (wonder already pins an absolute out-of-root path in the service env).
- Program.cs's pre-host ${secret:} expander now calls the FOUR-argument
AddZbSecrets with builder.Environment.ContentRootPath. This call site is
the actual fix, not the pin: the throwaway container has no
IHostEnvironment, and without the explicit root the under-content-root
rule is skipped at exactly the moment the migrator creates the store file
(the 0.6.0 defect). Do NOT simplify back to the 3-argument overload — it
compiles and silently reopens the gap. Pinned by a Program.cs source-scan
test AND a runtime falsification control that RUNS the migration and
demonstrably creates the store at the rejected path under the 3-arg form.
- docker-dev: Secrets__SqlitePath moves /app/data → /data via a SECOND mount
of the same per-node volume (same file at the volume root, so seeded
stores carry over) because /app is the container's content root and the
new rule correctly rejects it. LocalDb stays on /app/data (secrets-only
rule).
- SecretsStorePathValidationTests: positive-control probes, artifact-first
(relative and in-content-root paths must FAIL leaving no file; outside-
root passes; the no-explicit-root control proves the file assertions can
fail) — a clean boot is equally consistent with the validator never being
wired.
Live-verified on the six-node docker-dev rig (at 0.6.1): all nodes boot
under the new validation with stores at /data; an AdminUI-written secret
converged into BOTH central nodes' independent stores; site-a held only its
own pair's secrets (Phase 6 pair-local isolation intact). Replicator wire
behavior unchanged 0.3.0–0.6.x per CHANGELOG; registration-graph pins and
the full suite pass.
Production ran six weeks with a healthy process and a 0-byte log because
Serilog isolates per-sink failures: a file sink whose every write fails is
invisible from anywhere but SelfLog, which was not enabled. Two instruments:
- LogPipelineDiagnostics.EnableSelfLog (before the first logger builds):
SelfLog to logs/serilog-selflog.txt (rotated at 5 MB) + stderr — the only
channel that carries the exception the sink swallows.
- CheckFileSinkGrowth (after the startup banner): verifies each configured,
unbuffered file sink's rolled-file family physically grew across a marker
event; a silent sink is reported at boot via SelfLog/stderr and the
console sink instead of discovered months later. Growth-across-marker,
not existence — a previous run's bytes must not mask a silent sink (the
false pass an existence check would have given on wonder-app-vd03).
Never throws: both instruments swallow their own failures — a diagnostics
channel that can fail startup is worse than the defect it watches for.
The load-bearing test is the failure case (a sink that writes nothing IS
reported); a check that can only pass is indistinguishable from no check.
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and
LocalDbSyncAuthInterceptor all decided "is a key configured?" with
string.IsNullOrEmpty. A whitespace-only configured value therefore passed
that guard and fell through to the constant-time comparison — where it is
satisfied by a whitespace-only bearer token. The gate would be open to
anyone who guessed a blank credential.
This backports the check the two NEW mesh interceptors already use:
MeshCommandAuthInterceptor (and its central-side clone) validate with
string.IsNullOrWhiteSpace, so the four/five planes now answer the same way
to the same misconfiguration. Nothing else changes — the denial status
codes (PermissionDenied / Unauthenticated / PermissionDenied respectively)
and every denial message are untouched, and no key material is logged.
No deployment configures a whitespace key: the docker-dev rig and
production both set real keys, so this is a fail-closed hardening of an
unreachable-today misconfiguration rather than a live exposure.
Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied
fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied
(configured key " ", presented "Bearer "), verified red before the fix.
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the
HasExactlyOnePublicConstructor pin their two siblings already had —
Grpc.AspNetCore silently stops invoking an interceptor with more than one
public constructor, which would disable auth without failing anything.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:
- SetKnownClusters + InvalidateAll run on EVERY successful contact load
in EVERY mode, and the contact-refresh timer starts in all three.
Under Dps this is a ConfigDb query every ContactRefreshSeconds that
did not run before; central always has a ConfigDb, and one indexed
read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
filter moved out of the query's WHERE and into the loop, so the FAN
set is byte-for-byte what it was (enabled && !maintenance) while the
gate widens. This also dissolves the maintenance coupling flagged in
1A.2: a cluster whose nodes are all disabled or in maintenance can
still ack, and refusing it would make maintenance mode an
authentication switch. Central still refuses clusters it has never
heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
published, cache dropped — now pinned by test.
Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.
Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.
The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.
ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.
Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.
Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.
No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.
ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
Joseph Doherty