The hand-rolled bearer handler is replaced by AddJwtBearer (Microsoft.AspNetCore.Authentication.JwtBearer
10.0.7, pinned on the same servicing line as the other AspNetCore packages). Validation parameters come
from JwtTokenService.BuildValidationParameters, which now pins ValidAlgorithms = HS256 for both the
bearer scheme and TryValidate; MapInboundClaims is off so the minted Role/zb:username keys resolve.
/auth/token requires the cookie scheme, so a bearer can no longer mint a fresh token for itself.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- RunWriteAsync wraps the invoker call in WaitAsync(writeTimeout) (Props,
default 10 s = 2x the 5 s write cap): a token-ignoring hung write now
answers BadTimeout, releases its slot and tag, and its orphan outcome
is observed and discarded - later writes to that tag reach the device.
- WriteCompleted carries the launching incarnation (from the process-wide
generation source); a late completion from a previous incarnation after
a supervision restart is ignored instead of freeing the new
incarnation's slot/tag.
- Test: a hung token-ignoring tag-a write is abandoned and a later tag-a
write reaches the device while the abandoned one is still inside it.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- GatewayHistorianDataSource.ReadRawWindowAsync keeps the cap + 1 probe budget
(budgetCap) separate from the client's cap: a strict-ceiling or crowding
refusal now reports Limit = the server cap that applied and advises setting
NumValuesPerNode at or below it (it said "lower NumValuesPerNode" with
Limit = cap + 1 to a client that sent 0).
- Fuzz: the oracle-source over-cap assertion was vacuous (cap 0 never counts);
it now checks no reverse window reached the forward-only shape. Class docs
name the post-floor cap set (3 ... 1000) and distinguish NumValuesPerNode
1 / 2 from the smallest server cap.
- HistoryPaging.IsFullPage doc says it judges the REQUEST cap (page + ties +
lookahead), which is finite under any limit; AdvertiseMaxReturnDataValues
swallows any exception (returns false for the caller to log), as its doc
promised.
Refs #736
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
BoundedApiKeyStore's negative cache was keyed by key id, so a flood of DISTINCT well-formed
otua_<random>_x credentials missed it every time and cost one SQL query per attempt under the
SDK's SessionManager-wide lock, no credential needed (R03:3.3).
- IApiKeyStore.ListKeyIdsAsync (default null = unsupported; ConfigDbApiKeyStore lists every
ApiKeyId): the bounded store keeps a known-id set refreshed every StoreCacheSeconds
(single-flight, failed refresh keeps the previous set) and refuses an unknown id without a store
read ({outcome=unknown_id}).
- A process-wide miss budget (token bucket, Security:ApiKeys:MissBurst=100 /
MissRefillPerSecond=20, ApiKeyStoreGuardOptions) bounds lookups that reach the store and miss
while no set is available; a hit refunds its token. Per-endpoint accounting needs the
OpcUaServer impersonation seam to carry the client address — follow-up.
- docs/security.md: the new knobs, and the known-cluster gate's documented fail-open window at
central boot (R03:4.2) plus the async PSK refresh (R03:3.4).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- Writes: a second write to a tag already inside the driver waits in the
queue behind it (_tagsInFlight) while other tags still run under the
cap of 4; queued writes are stamped and failed BadTimeout at dequeue
once older than the 8 s host Ask budget (Props queuedWriteBudget)
instead of being written late.
- Initialise attempts are bounded pessimistically (initializeTimeout,
default 30 s = 2x Tier C) on top of the invoker's token-only timeout:
an overrun is logged as abandoned, reported InitializeFailed so the
retry loop continues, and its late result is dropped by generation.
Generations come from a process-wide Interlocked counter so a prior
incarnation's late result cannot collide (F6).
- PostStop shares one shutdownTimeout budget across shutdown + dispose.
- TelemetryDialSupervisor cancels the establishment deadline on
StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside
the try (a malformed node degrades to "unavailable"); the diagnostics
page labels the column "Last health publish".
- Tests: same-tag serialisation with a distinct-tag positive control;
stale queued write BadTimeout; token-ignoring hung init -> later
attempt connects; stopwatch assertion demoted to a ceiling.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- MeshGrpcCommandDialer.RetainOnly (IMeshCommandDialer, default no-op): CentralCommunicationActor
hands the dialer the current endpoint set on every contact refresh, so a renamed host or moved
port releases its cached GrpcChannel and idle HTTP/2 connection instead of living for the
process (R03:1.3).
- IMeshPskProvider.RefreshAsync (default = the old InvalidateAll): MeshPskProvider pre-resolves
every known cluster's key concurrently, bounded like the sync path, and swaps an immutable
snapshot in atomically; a failed read keeps that cluster's previous key. The actor calls it from
ContactsLoaded (after SetKnownClusters, off-actor via PipeTo) instead of InvalidateAll, so the
first ack per cluster per ContactRefreshSeconds no longer blocks a gRPC thread on a 3 s-bounded
store read or gets denied past the bound (R03:3.4).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
AuditWriterActor documented per-row dedup on the UX_ConfigAuditLog_EventId index but flushed one
statement batch: a single duplicate EventId (any at-least-once producer across a singleton
hand-over) threw for all 500 rows (R03:2.6). SaveChanges now detaches the entries a
DbUpdateException names (2601/2627 logged at Debug as the expected duplicate, anything else at
Warning) and re-saves, bounded by the batch size; a failure naming no entry stays a whole-batch
failure.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
(JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
/auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- NodeManagerHistoryReadRawCapFuzzTests: 10k seeds (env OTOPCUA_PAGING_FUZZ_ITERATIONS)
through a real SDK node manager against wire-faithful fakes of SqlServer / QuestDB /
Gateway that enforce the backend half of the cap; random series with ties, random
windows (empty, on/past a tie, between samples, forward/reverse/end-only, bounds),
N in {0,1,2,small,huge}, cap in {off,3..1000}, tie budget {3,8,65536}. Pages must
concatenate to the cap-off one-shot answer (checked against the series), terminate
within values + 2 (+ one empty step per oversized boundary cluster), never exceed
the cap, never ask the backend for 0 / over-cap; refusals only past the budget.
- HistoryPaging.AdvertiseMaxReturnDataValues writes HistoryServerCapabilities/
MaxReturnDataValues (i=11273), read back through the SDK master node manager.
- MinMaxRawValuesPerRead raised to 3: at 2 a resumed page has no room (value + its
boundary tie + lookahead) and every value cost an empty page + CP (seed 15).
- RawPageLimit rows pin the default-cap arithmetic (9 999 first page, 9 998 resumed).
Fuzz findings recorded as follow-ups: a bounded read whose data fits one page
returns cap + 1 (end bound fitted against N, not the cap; node manager, F2);
the Gateway walk-back's cost on a sparse series under a tiny tie budget is
pre-existing (seed 320: 82 reads uncapped, 93 capped).
Refs #736
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
A failed SDK start (port in use, PKI unwritable, bad OpcUa:* override) was one Error line: the
node stayed /health/ready 200, joined its mesh, could be elected driver Primary and acked deploys
through the Null sink while every client got connection-refused.
- IOpcUaEndpointState / OpcUaEndpointState: the start outcome is a fact the hosted service
maintains (NotStarted / Started / Failed / Stopped + reason).
- OpcUaEndpointHealthCheck ("opcua-endpoint", Ready + Active) is Unhealthy unless Started;
registered by AddOtOpcUaHealth(hasAdmin, hasDriver) on driver-role nodes.
- A driver-only node rethrows the start failure (Program.cs's fatal handler exits non-zero so
the supervisor restarts it); a fused node keeps its AdminUI up but is visibly not ready.
- StopAsync detaches the sinks and then STOPS the SDK (the port closes, clients get
connection-refused rather than a stale, inert endpoint); DisposeAsync is idempotent with it.
- The ServiceLevel calculator / redundancy election live in Cluster + Runtime (not this lane);
the fact is exposed for the Wave 3 follow-up to consume.
Tests: port-in-use ⇒ state Failed, probe Unhealthy, real /health/ready and /health/active 503
(positive control: 200 once Started); driver-only rethrows; StopAsync closes the port and
DisposeAsync twice is a no-op; the probe registration is on both tiers for driver nodes only.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceDiagnostics.Name carries the string driver-instance id
(what /hosts, telemetry and the artifact key on) and DriverInstanceId
the DB row Guid; the wire record has no slot for the authored display
name. The write-cap test asserts entry order per slot pair instead of
completion order of two concurrent writes.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverHostActor wraps the health publisher it hands its children in a
ChildHealthLedger (last snapshot per driver id, forwarded to the real
publisher), so GetDiagnostics now answers with each child's row id,
authored name, last-published state, device counts (host statuses when
the driver probes hosts, else one device) and last-change time — instead
of Guid.Empty / "Spawned" / 0 / now. Entries are forgotten on StopChild
and operator Restart.
IFleetDiagnosticsClient gains its first consumer: the new
/fleet/diagnostics page (node selector + ?node= query, revision,
running-from-cache notice, per-driver table). FleetDiagnosticsClient
logs the degrade-to-empty path at Warning.
Review ref: 02/U1.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
ReceiveAsync suspended the mailbox for the whole of every handler, so one Test Connect against a
black-holed PLC (60 s clamp) or one large deploy compose stalled every other admin operation
fleet-wide — AdminUI alarm acks/shelves, driver restarts, the next deploy.
- TestDriverConnect runs the probe on the thread pool and pipes a TestDriverConnectResult
straight to the requester (every path yields a result, never Status.Failure).
- StartDeployment serialises only the in-flight check (DB row + an actor-local compose-in-flight
flag, because the Deployment row is written at the END of the compose); the snapshot read,
validation, gzip and row write run off-actor and return as DeployComposed /
DeployComposeFailed, where the coordinator is told and the requester answered.
Tests: an alarm ack completes in < 1 s while a 60 s probe is pending and while a compose is
parked on a gated DbContext (positive controls: both still complete afterwards); a second
StartDeployment during an in-flight compose is refused.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
CallTimeoutSeconds was bound, defaulted and documented but consumed
nowhere. TelemetryDialSupervisor now arms a per-generation deadline on
every dial: a dial that has delivered no event by the deadline is
cancelled and treated as a transport failure (reconnect with a failure
streak). A live node replays its snapshots on subscribe, so this bites
only a dial that never connects (a black-holed host, where the socket
connect has no timeout). An established stream is deliberately not
bounded — the wire has no heartbeat, keepalive pings own its liveness.
Review ref: 02/U2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
StartDeployment read the whole config DB twice with no transaction — DraftSnapshotFactory for
the validator, then ConfigComposer for the artifact, plus clusters/nodes and Script.SourceCode
again in between. An AdminUI save landing between the two shipped an artifact the DraftValidator
never saw.
- DraftSnapshot carries ServerClusters, ClusterNodes and ApiKeys; DraftSnapshotFactory reads
every table inside one SNAPSHOT transaction on a relational provider (no-op on InMemory or
under an ambient transaction).
- ConfigComposer.Compose(DraftSnapshot) is pure over the snapshot (ordinal-sorted keys);
SnapshotAndFlattenAsync(db) delegates to it, byte-identical.
- AdminOperationsActor validates topology, estimates compile cost and composes the artifact
from the one snapshot it holds.
- Migration EnableSnapshotIsolation: ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON
(suppressTransaction; no model change).
Tests: the SQL Server snapshot read ignores a row committed mid-read (positive control: a later
read sees it); an edit injected between the Equipment and Tags reads is absent from the shipped
artifact and rejected by the next deploy; Compose(snapshot) == SnapshotAndFlattenAsync(db).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
- DriverHostActor.StopChild removes the driver's _lastReprovisionUtc stamp
(and its deferred-reprovision timer): the map grew per removed id and a
re-added driver inherited a backoff it never earned (02/L2).
- Stale now registers explicit drops for AttributeValueBatchPublished,
AttributeAlarmPublished, ArtifactForApply and ApplyWedged instead of
letting them hit Unhandled (02/H5).
- DbHealthProbeActor runs its SELECT on the thread pool and pipes the
outcome back (single-flight), with a 2 s command timeout on relational
providers, so a hung SQL Server no longer pins the actor thread and the
cached GetStatus answer stays reachable (02/P7).
- TelemetryLocalHub eviction tombstones are timestamped and swept past a
10 min TTL on the next Evict; a TimeProvider ctor is the test seam (02/L4).
- Tests: ValuePublishes.Single helper replaces the per-value
AttributeValuePublished in CalculationDependencyFlowTests (02/U4, test
half; the production receive is owned by Lane E).
- R05/8 audit: PollGroupEngine.Unsubscribe has no production caller; every
driver already uses UnsubscribeAsync (no change).
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
WriteAttribute was a ReceiveAsync, suspending the child's mailbox for the
whole device round-trip (up to 5 s): writes serialised, the second of two
operator writes to a slow device missed the host's 8 s Ask and was
reverted although the device took it, and every data change queued behind
the write. Writes are now a synchronous Receive that launches the device
call on the thread pool (sender captured, reply from the task, slot freed
by a self-sent WriteCompleted) under a per-driver cap (default 4; excess
writes queue in order). Queued writes fail fast on every exit from
Connected. Subscribe/unsubscribe stay ReceiveAsync.
Review ref: 02/P2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Review R08:1.1: an uncapped HistoryRead-Raw (NumValuesPerNode = 0) read a
tag's whole window into memory on every backend (SQL TOP(int.MaxValue),
QuestDB LIMIT, Gateway max_values = int.MaxValue + a second snapshot copy).
New ServerHistorian:MaxRawValuesPerRead (default 10 000; <= 0 unbounded with
a Validate() warning; 1 raised to 2). Each backend data source takes it,
reports it as IHistorianDataSource.MaxRawValuesPerRead (the Gateway: min
with a strict gateway's learned ceiling), so the node manager's existing
RawPageLimit paging serves N = 0 and N > cap as pages of at most the cap
with a continuation point. Independently, a backend raw read asking for 0
or more than the cap reads cap + 1 rows and is REFUSED
(HistoryReadLimitExceededException => BadTooManyOperations) when the window
holds more - never answered truncated, since the server's paging reads a
short answer as the last page. Bound through the Host registry, the
SqlServer/QuestDb providers and GatewayHistorian.CreateDataSource.
Tests: options (default/binding/warnings), provider binding, Gateway unit +
through a real SDK node manager (uncapped -> 999/998/998/605 + CPs), and a
conformance case for the SQL backends.
Refs #736
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
The Reconnecting retry timer started a new ShutdownAsync + InitializeAsync
pair on every tick whether or not the previous attempt had finished; the
generation token only discarded the older RESULT, so with an Initialize
bound longer than the cadence the teardown ran underneath an attempt still
inside the driver (the dispose-before-overwrite race). An _initInFlight
gate, cleared by the piped InitializeSucceeded/Failed, skips a tick while
an attempt is running; the pre-reconnect teardown is bounded by the
shutdown timeout; PostStop cancels an in-flight attempt before shutting
the driver down.
Review ref: 02/H2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
DriverInstanceActor.PostStop now bounds IDriver.ShutdownAsync by a
shutdownTimeout (default 5 s, logged on overrun) and then disposes the
driver (IAsyncDisposable preferred, else IDisposable) — but only on a
real stop. PreRestart flags the restart so the PostStop it triggers
skips the dispose, because a supervision restart re-uses the same
driver object (Props closure) and the host re-initialises it.
DiscoveryDriverBrowser disposes its throwaway capture/probe driver
after the bounded shutdown, including after a failed or timed-out one.
Review refs: 02/L1, 02/P4, 06/G-1 (host context), 07/L2.
Refs #731
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
A Deployment row left in Dispatching with no NodeDeploymentState rows (HandleDispatch threw
before its SaveChanges, or the process died between the AdminOps row write and the
coordinator's Tell) was recovered with an empty expected set; the deadline then hit the
0 == 0 "race" branch and returned without touching the row, so every later StartDeployment was
refused AnotherDeploymentInFlight until someone hand-edited SQL.
- PreStart: a recovered row with no expected-ack rows is a lost dispatch. Inside the apply
deadline it is re-driven through HandleDispatch (bounded: CreatedAtUtc never moves); past it
the row is closed TimedOut with a reason naming the condition.
- HandleDispatch is idempotent on a re-dispatch (rows already seeded ARE the expected set).
- HandleDeadline takes the race branch only when _expectedAcks.Count > 0.
- The PreStart recovery read projects ArtifactBlob away (R03:3.5).
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Keeps the root CLAUDE.md lean by lazy-loading per-area detail (drivers, topology,
data-plane auth, alarms, historian) only when matching files are touched.
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
Joseph Doherty
dohertj2