31 Commits

Author SHA1 Message Date
Joseph Doherty 2a8e2981fe feat(scripting): ScriptWorker isolation on Windows: restricted worker identity, pipe ACL, firewall + PeCacheDir installer (lmxopcua#780)
v2-ci / unit-tests (push) Has been cancelled
Owner decisions §14 #5/#6 as built (design §12.5):

- Scripting:Worker:Identity (default unset, Windows only, fail-closed validation): each worker
  starts under a restricted token derived from the host's. Every privilege but SeChangeNotify is
  removed; the restricting SIDs are NT SERVICE\OtOpcUaScriptWorker's SID + Everyone + Users +
  RESTRICTED; the token and default DACLs are host + SYSTEM + identity + RESTRICTED. The worker
  is created suspended, put in its Job Object before it runs, inherits only its stdio pipes, gets
  an ACL'd work dir as TEMP, and runs with DOTNET_EnableDiagnostics=0. The identity is granted on
  the host's window station and desktop, because USER32 init made coreclr.dll fail with
  ERROR_DLL_INIT_FAILED on windev. The logon SID is not used because the host's process DACL
  gives it PROCESS_VM_READ.
- The worker pipe on Windows always has an explicit protected DACL: host account + identity
  only. There is no owner ACE, so an elevated host's Administrators gets no access.
- The worker treats access denied on its parent-pid poll as alive; the Job Object covers that case.
- scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1 are idempotent
  and support -WhatIf, -Uninstall and -PlanOnly. They create the disabled anchor service, the
  PeCacheDir ACL (protected, no Users, key and markers host-only, pe\ Modify for the identity),
  read+execute on script-worker\, a ReadData deny on the install root, the outbound firewall
  block on the worker exe, and optionally the host-service env.
- Tests: the dry-run harness scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (46
  assertions, mutation-checked); ScriptWorkerIdentityTests; WorkerIsolationTests, which cover
  the pipe ACL, token access and a real restricted worker running a script, all green on windev.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 08:46:23 -05:00
Joseph Doherty 03686eb5d7 feat(scripting): Scripting:Runner=Shadow — in-process results published, each compiled run mirrored to the ScriptWorker pool and compared (lmxopcua#780 P2)
- ShadowScriptRunner (Core.Scripting): IScriptShadow.Mirror copies the inputs and queues the shadow run on the
  pool; compares value (structural), CLR type, status and failure class (compile / budget / threw+message /
  unsupported_type); runner-fault shadow outcomes (Unavailable, worker busy) are not compared. Bounded:
  Scripting:Shadow:MaxInFlight (256) and one per script; drops counted (behind | key_busy).
- Metrics otopcua.script.shadow.{compared,mismatch{script},dropped,inflight}; one rate-limited Warning per
  script and reason with a diff summary (Scripting:Shadow:MismatchLogsPerMinute, re-armed at the seam's deploy).
- The VirtualTag, Calculation and scripted-alarm seams recognise the shadow and keep their in-process path
  byte-for-byte (no coalescer, one-at-a-time pump, in-process engine), mirroring after each compiled run.
- ScriptRunnerSelection builds the shadow over an OutOfProcessScriptRunner whose lines carry
  ScriptRunnerRole=Shadow; Status() is null under Shadow so readiness never sees the shadow pool; the
  script-runner health entry stays Healthy and reports the shadow counters as data.
- docker-dev/docker-compose.script-runner-shadow.yml; docs/VirtualTags.md "Shadow mode".
- Tests: comparison table, per-script metric + once-per-reason log, rate limit, hung/blocking/throwing shadow
  never reaches the caller, MaxInFlight drop accounting, input copy; real-worker divergent + unsupported-type
  scripts counted against exactly those scripts, a worker-only runaway killed/quarantined there with every
  published value intact, alarm transitions identical to in process across a kill -9 of the shadow worker.
  Mutation-proved (16 mutants, 15 red; the one survivor added an assertion).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 07:23:06 -05:00
Joseph Doherty 0796928504 feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
  The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
  repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
  engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
  deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
  queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
  process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
  batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
  freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
  carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
  design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
  batched p99 <= 53 ms at 1,000.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 06:37:36 -05:00
Joseph Doherty 69a6e0b4c8 feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 23:33:08 -05:00
Joseph Doherty d683b83156 fix(scripting): give scripted-alarm predicates the configured run budget, not 250 ms (lmxopcua#779)
DriverHostActor built ScriptedAlarmEngine without a scriptTimeout, so every
production alarm predicate fell back to TimedScriptEvaluator.DefaultTimeout
(250 ms) while virtual tags and Calculation tags ran on 2 s. History shows no
decision behind it: 250 ms is the generic Phase-7 A.4 default, the VT
evaluator moved to 2 s in F8b (219d10a2) and the alarm engine was simply never
given a value.

- New node-wide Scripting:RunTimeout (default 2 s, clamped to (0, 60 s],
  non-positive -> default, each clamp a startup Warning) on ScriptingOptions.
- The VT evaluator and the alarm engine default to it; DriverHostActor now
  builds the engine through CreateScriptedAlarmEngine, which passes it
  explicitly. Calculation keeps its per-driver runTimeoutMs.
- Docs: ScriptEditor (config table), ScriptedAlarms, VirtualTags.

Tests: ScriptedAlarmEngineBudgetTests (a 600 ms predicate now completes and
activates; default and configured budgets), ScriptedAlarmEngineBudgetWiringTests
(the host-built engine), ScriptingOptionsBindingTests (bind + clamps). The
engine and wiring tests fail with the old 250 ms default restored.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:44:35 -05:00
Joseph Doherty 0702b31029 fix(alarms): refuse a keyed native ack with no Galaxy identity synchronously, from a cached key set (lmxopcua#765)
v2-ci / unit-tests (push) Waiting to run
Item 3 of #765. A keyed (API-key) session whose key had no Galaxy ack identity
got Good and then a revert, because whether the key has an identity lives in
the Secrets store, which must not be read under the SDK lock.

GalaxyAckIdentityDirectory (Driver.Galaxy) keeps a node-wide set of the key ids
that have a galaxy-ack-user secret. It lists this node's Secrets store
(metadata only, nothing decrypted) and is read lock-free.
GalaxyDriver.GetAcknowledgeRefusal (IAlarmAcknowledgePreflight, published
through NativeAlarmAckPreflightView as in #760) now refuses a keyed session
whose key the directory reports Absent, when the driver has no fallback ack
account. A refusal also asks for a refresh.

Fails open: before the first refresh, once the snapshot is older than
StaleAfter, and from a signalled identity-secret change until the next
refresh, the lookup answers Unknown and the ack takes the old Good-then-revert
path.

Refresh: GalaxyAckIdentityRefreshService runs on driver nodes, every
Drivers:Galaxy:AckIdentityCache:RefreshIntervalSeconds (default 30;
StaleAfterSeconds 0 = three intervals; validated at start). It also refreshes
early on secret changes. The Secrets library has no change event, so
SecretsRegistration decorates ISecretCacheInvalidator, which is called by
AdminUI /api-keys, /admin/secrets and the replication appliers. API-key
changes need no event: a revoked key is already denied before the pre-check.

Tests: Galaxy.Tests GalaxyAckIdentityDirectoryTests (16) and Host.Tests
GalaxyAckIdentityCacheWiringTests (9). Seven separate mutations each turn at
least one test red: pre-check ignores the directory, factory drops it, no
invalidator decoration, no staleness check, no change-pending check, no
refresh on refusal, loop never wakes early.
Docs: Galaxy.md, AlarmTracking.md, Configuration.md, .claude/rules/alarms.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:38:57 -05:00
Joseph Doherty 97dc715dac fix(runtime): derive every write bound and Ask from one write timeout (lmxopcua#777)
v2-ci / unit-tests (push) Has been cancelled
Three write bounds were configured independently: the inbound-write
gateway's 10 s Ask, DriverHostActor's fixed 8 s Ask of the child, and the
child's own write timeout (5 s + up to 2 s grace) with a fixed 8 s queue
budget. An outer Ask that fires first reports the write failed and the node
reverts while the child is still driving the device, so any write timeout
above ~6 s (or a write dequeued late in its 8 s budget) let the device take
a value the client was told failed.

DriverWriteBounds (Runtime) derives them all from DriverWrites:Timeout:
  token cancel   = Timeout
  abandon        = Timeout + min(Timeout/2, 2 s)            (RunBound)
  queue budget   = RunBound
  child replies  <= queue budget + RunBound                  (ReplyBound)
  host Ask       = ReplyBound + 1 s
  gateway Ask    = host Ask + 1 s
Default Timeout is 10 s (the Tier C Write timeout, so no tier default is
capped below its own policy); host Ask 25 s, gateway Ask 26 s. Tier A/B
writes are still cancelled first by their 2 s / 4 s tier timeouts.

- DriverHostActor takes writeBounds + writeClock, spawns children with them,
  and runs its Ask deadline as a token on the write clock (Infinite Akka
  timeout), so host and child bounds share one clock.
- WriteAttribute gains IssuedAt (write-clock timestamp, stamped by the
  host): the queue budget counts mailbox time too, and a write past it is
  failed BadTimeout and never started - including on arrival. Stale queued
  writes are expired across the whole queue, not only up to a free slot.
- ActorNodeWriteGateway's default Ask = Default.GatewayAsk; the hosted
  service passes the configured one.
- Startup validation (Host DriverWriteBoundsBinding, same warn-and-clamp
  style as ScriptingOptionsBinding): unparsable / non-positive -> default,
  above 5 min -> clamped, and each tier whose default Write timeout exceeds
  the write timeout is named in a Warning (its policy would never take
  effect). The derived bounds are logged at Information. Per-instance
  ResilienceConfig Write overrides arrive with a deployment, so
  DriverCapabilityInvokerFactory warns at spawn (writeTimeoutCeiling).
- appsettings.json documents DriverWrites:Timeout.

Tests (manual clock):
- DriverHostActorWriteAskTests: a held write under a 9 s timeout survives
  an 8.5 s step and succeeds; a token-ignoring write is answered by the
  child's BadTimeout at 9 s + grace, not by the Ask; the Infinite-timeout
  Ask ends exactly at its clock deadline.
- DriverInstanceActorConcurrentWriteTests: a write issued longer ago than
  the queue budget is failed and never reaches the driver; positive control
  at exactly the budget is written.
- DriverWriteBoundsTests: every outer bound covers the inner ones for 0.5 s
  to 300 s timeouts.
- DriverWriteBoundsBindingTests: default fits every tier; 3 s timeout is
  caught at startup with Tier B and C named (not A); bad values fall back
  with a warning; over-max clamps.
- DriverCapabilityInvokerFactoryTests: spawn warns when a Write override
  exceeds the ceiling; no warning within it or without a ceiling.
Mutation checks: a fixed 8 s host Ask on the same clock fails both 9 s host
tests; stamping arrival time instead of IssuedAt fails the issued-at test;
dropping the tier check / factory check fails their tests.

Runtime.Tests 961 passed / 3 skipped at AKKA_TEST_TIMEFACTOR 3 and 1;
Host.Tests 232/232 at 3 and 1; Core.Tests 361 passed / 1 skipped.
Solution Release build: 0 warnings.

ContinuousHistorizationRecorderTypedValueTests (historian) deliberately
left for the historian lane.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:46:30 -05:00
Joseph Doherty 4e76eeb57f test(historian): Gateway advertises TimeAverage/Total only behind EnableNativeTimeWeighted (lmxopcua#746)
HistorianBackendRegistryTests (#682) still expected the Gateway data source
to advertise the two AVEVA-native aggregates. Lane H2 (R08:2.1) refuses
them by default until historiangw#27, and the lane branch never updated
this Host test, so it failed after the rebase. The default case now expects
the ten raw-evaluated aggregates, and a new case pins that
ServerHistorian:Gateway:EnableNativeTimeWeighted=true, resolved through the
Host's registry, brings TimeAverage and Total back.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:40:47 -05:00
Joseph Doherty 4d8de20050 test(scripting): pin the Scripting section bind behind one helper
Review m-1. The bind moves from Program.cs into
Host.Configuration.ScriptingOptionsBinding.Bind, which binds the
section and clamps it. Program.cs calls it, and
ScriptingOptionsBindingTests drives it with in-memory configuration.
Every documented key takes a non-default value, so a misspelled
property or key goes red instead of silently falling back to the
defaults. The tests also cover a clamped bind (1e300 / 1024 / 0, three
adjustments), an absent section, and the sibling LogFilePath /
LogTopicMinLevel keys leaving the bind untouched.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 15:14:02 -04:00
Joseph Doherty f72af76e99 fix(security): validate the API-key pepper at start, finish JWT bearer, circuit-aware audit actor
- ApiKeyOptionsValidator (ValidateOnStart on any driver/admin node): a blank or placeholder
  Security:ApiKeys:Pepper fails host start instead of rejecting every key after a green boot (R03:2.3).
- JWT is finished and opt-in by key (R03:2.4): with Security:Jwt:SigningKey set, a Bearer scheme
  (JwtBearerAuthenticationHandler over JwtTokenService.TryValidate / BuildValidationParameters, fail
  closed) accepts the tokens /auth/token mints; the default scheme is a cookie-or-bearer policy scheme
  and the fallback policy names it, so /api/* and every RequireAuthorization endpoint take a browser
  session OR a machine token. JwtOptions are validated at start (JwtOptionsValidator). Without a key,
  /auth/token is not mapped (404, not the former constructor 500) and no bearer scheme exists.
  A direct AddJwtBearer needs Microsoft.AspNetCore.Authentication.JwtBearer in
  Directory.Packages.props (not this lane's file) — the handler is a drop-in to swap.
- AuthenticationStateAuditActorAccessor is the registered IAuditActorAccessor: the Blazor circuit's
  principal first, HttpContext as the fallback (R03:2.5). HttpAuditActorAccessor shares its resolver.
- CookieAuthenticationStateProvider (unregistered, fire-and-forget ping loop) is deleted (R03:1.5/2.5).
- /auth/login logs the LDAP exception at Warning before answering 503 (R03:2.7).
- Admin operations reply with a fixed message + correlation id instead of ex.Message (R03:4.1).

Refs #733

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 10:23:14 -04:00
Joseph Doherty 78f604e0cd feat(historian): Gateway serves the per-sample aggregates from raw reads with the shared Sql.Common evaluator (#681)
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).

- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
  row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
  look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
  historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
  BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
  caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
  the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
  carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
2026-10-12 09:56:54 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty f292cf3beb fix(host): #678.6 build the role-overlay suffix from plain roles only; audit overlays; driver overlay Opc.Ua Debug -> Information 2026-10-11 10:47:01 -04:00
Joseph Doherty 13d628a07f feat(historian): QuestDb provider, registry + consumer-gated validator, compose env
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:23:21 -04:00
Joseph Doherty 7500106c2f feat(historian): SqlServerHistorianProvider + SqlServer backend registration
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:42:38 -04:00
Joseph Doherty 31d342eff3 feat(historian): backend registry, ServerHistorian:Backend, Host wiring via provider
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:05:53 -04:00
Joseph Doherty 341417f193 fix(ab): share status map + libplctag orphan release, harden confinement guard, pin width-mismatch + DefaultTransport wiring (#568 #569 #580 #585)
- Ab.Transport: AbOpcUaStatusMap is the single AbStatus->OPC UA table; AbCip/AbLegacy mappers delegate.
- Remove dead MapCipGeneralStatus / MapPcccStatus / AbLegacy MapException (+ tests).
- LibplctagOrphanRelease is one source file (Ab.LibplctagShared) compiled into both drivers.
- LibplctagConfinementGuardTests: tokenising comment/literal stripper, block comments, csproj PackageReference guard.
- LibplctagStatusTranslationTests: real LibPlcTagException round trip, WrapSync/Wrap OCE pass-through, adapter routes decode via WrapSync.
- AbCipTypeMismatchTests: libplctag width-mismatch path (BadTypeMismatch, handle kept, no Degraded, warn once).
- Host: AddAbCipDriverDefaults extension + Program.cs source pin for Drivers:AbCip:DefaultTransport.
- docs/drivers/AbCip.md: layout rows + benign Degraded-after-outage note.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-07 22:59:06 -04:00
Joseph Doherty 2cea1a0687 fix(ab-transport): final review I2 — wire DeviceSessionPool.Shared to the host logger
ConfigureSharedLogger had no production caller, so the pool's only
diagnostics (first-wins options conflicts, scheduler/close failures)
went to NullLogger. DriverFactoryBootstrap now configures it from the
driver registry factory and from the AbCip/AbLegacy probe factories
(admin-only nodes reach the pool only through Test Connect). Wiring tests
reset the pool to a null logger first, so removing the call goes red
(verified).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-07 21:36:37 -04:00
Joseph Doherty b75e013d0a feat(abcip): final review I1 — node-level Drivers:AbCip:DefaultTransport
The old AdminUI cannot author the per-driver transport key (#576), so the
'pin before upgrading' recipe had no working path. A node-level appsetting
(env Drivers__AbCip__DefaultTransport) now supplies the default for AbCip
drivers with no key: explicit key > node default > compiled Native. Bound
and validated unconditionally (bad value fails host start); the driver
factory and the Test Connect probe both read it so they agree. AbCip.md
upgrade recipe rewritten around the key; Configuration.md documents it.
Also documents first-wins for AllowPacking/probe and host-alias throttles
(final review M6) and moves the status-bit note out of AbCip.md (M5).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-07 21:34:48 -04:00
Joseph Doherty 894444376e feat(security): register TimeProvider + S18/S19 option knobs (archreview 03/S18,S19) 2026-08-20 09:36:14 -04:00
Joseph Doherty b037cc7ac4 chore(secrets): adopt ZB.MOM.WW.Secrets 0.6.2 — fail-closed store-path validation
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 15m19s
Four-minor jump 0.2.3 → 0.6.2. The release line exists because a deploy
renaming the app directory took the MxGateway credential store with it
(2026-08-09, two silent days): Secrets:SqlitePath must now be absolute AND
outside the content root, rejected at boot rather than relocated. 0.6.2 over
0.6.1 is message-only (the rejection text now teaches the mounted-volume
case and forecloses the override hunt) — the live rig evidence below was
gathered at 0.6.1 and the validator behavior is identical.

Adoption changes beyond the pins:

- appsettings.json drops the relative "otopcua-secrets.db" (now boot-fatal);
  dev gets the library's absolute per-user default, deployments supply their
  own (wonder already pins an absolute out-of-root path in the service env).
- Program.cs's pre-host ${secret:} expander now calls the FOUR-argument
  AddZbSecrets with builder.Environment.ContentRootPath. This call site is
  the actual fix, not the pin: the throwaway container has no
  IHostEnvironment, and without the explicit root the under-content-root
  rule is skipped at exactly the moment the migrator creates the store file
  (the 0.6.0 defect). Do NOT simplify back to the 3-argument overload — it
  compiles and silently reopens the gap. Pinned by a Program.cs source-scan
  test AND a runtime falsification control that RUNS the migration and
  demonstrably creates the store at the rejected path under the 3-arg form.
- docker-dev: Secrets__SqlitePath moves /app/data → /data via a SECOND mount
  of the same per-node volume (same file at the volume root, so seeded
  stores carry over) because /app is the container's content root and the
  new rule correctly rejects it. LocalDb stays on /app/data (secrets-only
  rule).
- SecretsStorePathValidationTests: positive-control probes, artifact-first
  (relative and in-content-root paths must FAIL leaving no file; outside-
  root passes; the no-explicit-root control proves the file assertions can
  fail) — a clean boot is equally consistent with the validator never being
  wired.

Live-verified on the six-node docker-dev rig (at 0.6.1): all nodes boot
under the new validation with stores at /data; an AdminUI-written secret
converged into BOTH central nodes' independent stores; site-a held only its
own pair's secrets (Phase 6 pair-local isolation intact). Replicator wire
behavior unchanged 0.3.0–0.6.x per CHANGELOG; registration-graph pins and
the full suite pass.
2026-08-11 09:06:43 -04:00
Joseph Doherty f10f4c3754 fix(auth): #543 whitespace-only configured key fails closed in the three pre-existing gRPC auth interceptors
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and
LocalDbSyncAuthInterceptor all decided "is a key configured?" with
string.IsNullOrEmpty. A whitespace-only configured value therefore passed
that guard and fell through to the constant-time comparison — where it is
satisfied by a whitespace-only bearer token. The gate would be open to
anyone who guessed a blank credential.

This backports the check the two NEW mesh interceptors already use:
MeshCommandAuthInterceptor (and its central-side clone) validate with
string.IsNullOrWhiteSpace, so the four/five planes now answer the same way
to the same misconfiguration. Nothing else changes — the denial status
codes (PermissionDenied / Unauthenticated / PermissionDenied respectively)
and every denial message are untouched, and no key material is logged.

No deployment configures a whitespace key: the docker-dev rig and
production both set real keys, so this is a fail-closed hardening of an
unreachable-today misconfiguration rather than a live exposure.

Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied
fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied
(configured key "   ", presented "Bearer    "), verified red before the fix.
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the
HasExactlyOnePublicConstructor pin their two siblings already had —
Grpc.AspNetCore silently stops invoking an interceptor with more than one
public constructor, which would disable auth without failing anything.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:14:36 -04:00
Joseph Doherty 4d2dfb5b50 test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:

  1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
     point, kept as a test rather than a live-gate note: each cluster stays single-member and
     none ever holds another's address as a member.
  2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
     site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
     would be refused at the far side.
  3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
     it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
     process did". The same node then accepts the right key, so the silence is the auth gate
     and not a dead endpoint.

All three run fully offline — no Docker fixture — and pass.

Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.

The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
2026-08-06 15:58:36 -04:00
Joseph Doherty 027f0403c4 test(mesh): role-scoping scan handles braceless if (hasAdmin) — closes the phantom-block false pass 2026-08-06 10:33:15 -04:00
Joseph Doherty 2e06dfe52d test(mesh): pin the admin-only role scoping of IMeshPskProvider/IMeshCommandDialer registrations (security register item 3) 2026-08-06 10:22:29 -04:00
Joseph Doherty 48dc2120c6 fix(mesh): gate follows the endpoint not the mode + review fixes
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:

- SetKnownClusters + InvalidateAll run on EVERY successful contact load
  in EVERY mode, and the contact-refresh timer starts in all three.
  Under Dps this is a ConfigDb query every ContactRefreshSeconds that
  did not run before; central always has a ConfigDb, and one indexed
  read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
  filter moved out of the query's WHERE and into the loop, so the FAN
  set is byte-for-byte what it was (enabled && !maintenance) while the
  gate widens. This also dissolves the maintenance coupling flagged in
  1A.2: a cluster whose nodes are all disabled or in maintenance can
  still ack, and refusing it would make maintenance mode an
  authentication switch. Central still refuses clusters it has never
  heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
  published, cache dropped — now pinned by test.

Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.

Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.

The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.

ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
2026-08-06 06:44:20 -04:00
Joseph Doherty 14d21496fe feat(mesh): known-cluster gate + Matches on the PSK provider
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.

Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.

Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.

No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.

ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
2026-08-06 05:31:08 -04:00
Joseph Doherty 24e55d3395 fix(mesh): Batch B review fixes — bounded store read, cache poison guard, test hardening 2026-08-06 04:10:03 -04:00
Joseph Doherty 68a416e4fc feat(mesh): per-cluster PSK provider + central interceptor 2026-08-06 03:44:23 -04:00
Joseph Doherty fbda015c6e feat(mesh): node-side PSK interceptor 2026-08-06 03:37:02 -04:00
Joseph Doherty 50f1620d79 feat(mesh-phase5): fail-closed bearer interceptor for the telemetry stream
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 15:58:45 -04:00