Commit Graph

137 Commits

Author SHA1 Message Date
Joseph Doherty a4806576af feat(pending-updates): short-circuit an unchanged fleet on a DB rowversion watermark (#62)
Owner decision (b), 2026-11-09. PendingUpdatesEvaluator re-flattened every
deployed instance on each cache-TTL expiry even when nothing had changed.

- Migration AddConfigGraphRowVersions: a non-concurrency rowversion column
  + index on the 18 configuration tables the evaluation reads (Templates +
  5 member tables, Instances + 4 override/binding tables, DataConnections,
  DeployedConfigSnapshots, Sites, SharedScripts, ExternalSystemDefinitions,
  ExternalSystemMethods, DatabaseConnectionDefinitions). SQL Server only in
  the model; SQLite test contexts are untouched. Script:
  docs/plans/sql/AddConfigGraphRowVersions.sql. Joins the coordinated
  central-pair migration window (#107 item 4).
- IConfigGraphWatermarkReader / ConfigGraphWatermarkReader: ONE query of
  MAX(RowVersion) + COUNT_BIG(*) per table (count catches deletes, which
  raise no rowversion). rowversion is engine-bumped for any writer, so a
  peer central node's edit moves it — not the per-process watermark MG-3
  removed.
- The secret store (outside the DB) gets its own stamp: SHA-256 over
  ListAsync(includeDeleted: true) metadata.
- Evaluator re-serves the last CLEAN snapshot while watermark + secret
  stamp are unchanged; every PendingUpdatesFullEvaluationEvery'th (default
  10, validated >= 1, 1 = off) is full regardless — the safety net for a
  same-KEK re-wrap or an input table missing from the set. Always full when
  the token is unavailable/unreadable, after an evaluation that could not
  compute a site's artifact hash, on Re-check, and after a deploy completion.
  It only re-serves a real earlier answer, never manufactures one.

Tests: SQL Server (real migrated DB) — token stable without writes, moves
for a peer-connection INSERT/UPDATE and a non-max DELETE, moves for a local
EF write, a detached Update still saves (not a concurrency token), and a
guard that every table the evaluator's repository reads touch is covered.
Evaluator — reuse without flatten (re-served answer still shows the stale
instance), watermark move, 4 secret changes, every-Nth, 5 no-token modes,
unclean evaluation not reused, cache Re-check and deploy completion force
full. Mutation-proven: token compare, clean check, cache force flag, row
count, and an uncovered read each turn tests red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 22:38:55 -05:00
Joseph Doherty 63ec45e2e3 fix(flow-debugger): refuse method-entry Live runs for site-scoped users (#55)
Owner decision (b), 2026-11-09. An inbound API method has no site
affinity, so the F2 route-time scope only bounded routed SITE traffic;
the method's central-side effects (Database.*, ExternalSystem,
notifications) still ran for a scoped Deployer. FlowDebugRunner now
refuses FlowDebugRunMethodCommand for a site-scoped caller (site ids,
not Administrator) before the script executes, audits the refusal as a
failed InboundRequest under the operator's name, and raises
SiteScopeViolationException -> ManagementUnauthorized, so the Central
UI and the CLI (same /management command) share the gate.

The page disables the "Inbound API method" entry for scoped Deployers
with the reason (tooltip + inline note), starts them on script entry,
and ignores ?methodId= / a stored method choice. Replay and the
instance-scoped script / value-change Live runs are unchanged. The
route-time scope is kept as defence in depth.

Tests: scoped refused (in- and out-of-scope routing; script never runs,
refusal audited), unscoped/Administrator allowed, scoped Replay allowed,
scoped in-scope value-change Live still runs; bUnit for the disabled
entry, the drill-in, and the unscoped control. Mutation-proven (gate
and UI flag reverted -> 4 red).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 22:22:44 -05:00
Joseph Doherty 3ff463b6fd docs(audit): Parquet cold archive shipped, default off (ScadaBridge#42)
Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:12:12 -05:00
Joseph Doherty ab561faa43 feat(cluster): opt-in leader-auto-down strategy - in-repo port of Akka AutoDowning (ScadaBridge#106)
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET
1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the
leader among the reachable members downs a peer unreachable for StableAfter; a
non-leader records it as pending and downs it on becoming leader; reachable-again
or removed members are forgotten; Down/Exiting members are skipped. Its window is
carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no
deprecated auto-down-unreachable-after key is emitted and Akka's boot-time
"keep-majority will be used instead" warning goes away. DownRemovalMargin reads
akka.cluster.down-removal-margin exactly as AutoDowning does (default off).

New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon
and ClusterOptionsValidator. The default stays "auto-down"; auto-down and
keep-oldest HOCON is byte-identical to before. Placed in Host, not
ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays
Akka-free).

Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec
cases (leader-only, pending-on-leader-change, losing leadership, reachable again,
removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests
clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node
cluster plus a test that Akka actually loaded the provider and its downing actor.
Mutation-checked: a provider that never calls Cluster.Down fails both failover
tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed
mutations each fail the unit tests.

Remaining gate before switching the default: the docker failover drill
(DRILL_MODE=active and standby mode) on leader-auto-down.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:46:58 -05:00
Joseph Doherty dd99a9f5a5 docs(audit): hash-chain tamper evidence shipped, default off (ScadaBridge#41)
Component-AuditLog Security & Tamper-Evidence describes the per-month chain,
the chaining job and its lock, both purge paths (checkpoint anchor,
tombstones), pre-existing rows, verification and what an unsigned chain does
not prove. alog.md §11.4/§16, CLAUDE.md, HighLevelReqs AL-11/12,
Component-CLI, Component-ManagementService, docs/components/CLI.md and the
CLI README drop the "no-op placeholder" wording.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:33:11 -05:00
Joseph Doherty 47f3a703bf perf(search): prefix-by-default Deployments and config-audit search; bounded Contains opt-in (ScadaBridge#50)
Deployments free-text search and the config-audit EntityName filter no longer
emit LIKE '%term%' by default. Prefix (LIKE 'term%') is the default; substring
is an explicit opt-in (DeploymentSearchMode.Contains / entityNameContains)
always bounded by a DeployedAt/Timestamp window (explicit, else 30 days).
Migration PrefixSearchIndexes adds DeployedBy, RevisionHash and EntityName
indexes. Deployments page gains a 'Match anywhere' checkbox.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:42:56 -05:00
Joseph Doherty 69a2017000 docs(components): reference docs for ScriptAnalysis, KpiHistory and DelmiaNotifier (ScadaBridge#72)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:02:49 -05:00
Joseph Doherty 7892c622cf fix(health): site /health/active uses the shared role-scoped ActiveNodeHealthCheck (ScadaBridge#97)
Replace the private SitePairActiveNodeHealthCheck (503 whenever no ActorSystem, no data)
with the shared ZB.MOM.WW.Health ActiveNodeHealthCheck constructed with the site-{SiteId}
role: Degraded (200) during startup per the Health spec, and it publishes
selfAddress/activeRole/activeNode so the Overview dashboard can name the active site node.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 14:13:53 -05:00
Joseph Doherty 23c35df234 refactor(inbound-api): use the shared ZB.MOM.WW.Health IActiveNodeGate (ScadaBridge#80)
The InboundAPI carried its own IActiveNodeGate (implemented by Host/Health/ActiveNodeGate)
whose rule already matched the shared Health 0.3.0 AkkaActiveNodeGate — both route to
ClusterActiveNode (oldest Up member). The private interface and implementation are deleted;
InboundAPI references ZB.MOM.WW.Health (0.3.0, already pinned) and InboundApiEndpointFilter,
AuditWriteMiddleware, InboundMethodWarmupService and FlowDebugRunner consume
ZB.MOM.WW.Health.IActiveNodeGate. The Central branch of Program.cs registers
AkkaActiveNodeGate (unscoped, like the active-node health check).

Warm-up behaviour is kept: the shared gate answers false while no ActorSystem resolves, while
this member is not Up, and on any cluster-init fault. It resolves the ActorSystem through the
existing DI singleton bridge (GetOrCreateActorSystem), the same path the health probes use.

Tests: CompositionRoot asserts the shared gate type (fails with the registration removed);
the evaluator wiring test now drives AkkaActiveNodeGate; new warm-up test (no ActorSystem ->
not active). Nothing was missing from the Health 0.3.0 package.

(ScadaBridge#80)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:54:29 -05:00
Joseph Doherty 5310fd0e26 feat(streaming): carry IsMesReportable on AlarmStateUpdate (25) and DebugAlarmStateDto (20) (ScadaBridge#60)
Central's Debug View live stream and Alarm Summary cache always saw false
because neither wire message had the field. Additive proto fields, vendored
C# regenerated with docker/regen-proto.sh (sitestream + sitecommand), mapped
both ways in StreamRelayActor/SiteStreamGrpcClient and SiteCommandDtoMapper.
The live stream publishes Alarm Actor events, which never carried the flag,
so InstanceActor.HandleAlarmStateChanged now stamps it before retaining and
publishing. A pre-#60 site sends the proto3 default. No UI badge yet.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:31:37 -05:00
Joseph Doherty 0e89966da6 docs: point markdown backlogs at Gitea issues (status lives in Gitea) 2026-11-06 15:31:44 -05:00
Joseph Doherty 3e1435dfac docs: QA central pair relays standby inbound API calls (StandbyForwardPeerEndpoints) 2026-10-22 16:59:07 -04:00
Joseph Doherty 4bc124d73e docs: QA central pair moved to shared SQL key + secret stores; record the Windows import-rename defect 2026-10-22 15:56:25 -04:00
Joseph Doherty 3808b49fc7 docs: record area-subtree deploy, Topology preload and the #38 rename trap in CLAUDE.md; fix stale deploy-site client-timeout line in the CLI README 2026-10-22 14:37:16 -04:00
Joseph Doherty 2acd14f276 docs: CLAUDE.md review-remediation facts + 2026-10-21 remediation record 2026-10-22 09:51:52 -04:00
Joseph Doherty 55e1741075 fix(site-secrets): prune secret_mirror_status rows of decommissioned or renamed nodes
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval
(10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and
deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the
refresh). The delete is conditional on the checked_at it read, so a node that came back
keeps its fresh row; a returning node re-writes its row on its first pass.
2026-09-29 10:56:53 -04:00
Joseph Doherty 4e6409184c Merge branch 'fix/standby-secret-mirror-status' (M1: standby mirror failures on the health report)
# Conflicts:
#	CLAUDE.md
#	docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md
2026-09-28 21:37:03 -04:00
Joseph Doherty f5756affef docs(site-secrets): document secret_mirror_status and standby mirror failures on the health report (M1) 2026-09-28 21:33:10 -04:00
Joseph Doherty 29049c4ef6 fix(site-secrets): contain secret-store outages per site; audit removal-only deploys
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
2026-09-28 21:23:52 -04:00
Joseph Doherty d641186397 docs(site-secrets): final-review doc fixes — replication prerequisite, upgrade impact, audit shape
I2: topology guide + Component-SiteRuntime + register row 127 state that a site pair using
deployed secrets must run LocalDb replication. I3: row 127 + topology guide state the upgrade
impact (whole artifact deploy fails and reads Out of date until referenced secrets exist
centrally). M6: audit row carries secrets/missing, no removed field; removal-only deploys write
no central row. M2/M3/I1 as-built notes; CLAUDE.md secrets bullet corrected.
2026-09-28 21:08:34 -04:00
Joseph Doherty 0ccf68951d docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge 2026-09-28 20:22:45 -04:00
Joseph Doherty b3021e7c5f feat(scripts): read-only, allow-listed Files global for site scripts; Delmia test receiver parses recipe files
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.

Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
2026-09-28 10:36:42 -04:00
Joseph Doherty fe89363b1a docs(claude): winrm re-enabled for deployment activities only (2026-09-28)
The 2026-09-11 hold (WinRM exec tripped the wonder boxes' security software) is lifted for
deploys only: preflight, backup, stop/swap/start, migration, post-deploy verify, rollback.
Ad-hoc diagnostics and deployed-version checks use SMB/HTTP/SQL instead. psbridge stays retired.
2026-09-28 09:25:10 -04:00
Joseph Doherty d8979664f1 docs: site artifact revision, pending-updates indicator, rail scroll pane
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
2026-09-14 14:01:07 -04:00
Joseph Doherty ce8fc092c8 docs: MES-reportable alarm flag — component docs, CLAUDE.md, register rows 121–122
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
2026-09-11 10:17:02 -04:00
Joseph Doherty da0084886a docs: note that psbridge and winrm remote access to the wonder boxes are disabled (2026-09-11)
Claude-Session: https://claude.ai/code/session_01PpXxRCFznFqE65ay6wAJwe
2026-09-11 09:05:36 -04:00
Joseph Doherty ae23c40a74 docs: reset-aware handshake — spec note, component doc, Delmia/Ipsen notes, register row 119, CLAUDE.md (#36)
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
2026-09-10 15:23:09 -04:00
Joseph Doherty b5eb08f1a9 fix(flow-debugger): method pane takes structured parameters via ParameterValueForm + raw-JSON escape hatch (#37)
The Run-inbound-API-method pane rendered one text box per top-level schema
field and coerced the text on the page; an array/object field went through
JsonNode.Parse and, on failure, was SENT AS A STRING, while an empty box was
silently dropped. A required array-of-object parameter (MesMoveOut WorkOrders)
could not be entered at all — QA runs reached the site with "WorkOrders":[].

- The method pane now renders the same shared ParameterValueForm the script
  pane uses (typed inputs, nested groups, add/remove list item editors,
  $ref:lib: resolution, inline validation), keyed on the method id so a change
  of method remounts it. The page keeps ONE value tree and BuildParameters()
  serializes it as-is; the flat _inputs / Coerce() / SetInput path is deleted.
- Form | Raw JSON toggle: raw mode is a Monaco json editor for the whole
  request body, given the method's root schema for completion, parsed and
  schema-validated (InboundApiSchema.Validate) on every edit with the error
  inline. Entering raw mode seeds the editor from the tree; leaving it
  re-mounts the form over the last valid tree.
- Run is disabled while the form or the raw document is invalid (a required
  field left empty disables Run before the first keystroke).
- localStorage per-method entry stores the value tree; on restore it is
  validated against the schema and dropped if invalid, which also retires the
  pre-#37 flat string-per-field entries.
- ParameterValueForm.ToJsonSchema / JsonElementToObject made internal static
  for page reuse.
- Tests: 7 new bUnit tests (WorkOrders array-of-object round-trip to the
  RunMethodAsync payload, required list-item gating, invalid raw JSON blocks
  Run + shows parse/schema/non-object errors, valid raw JSON sent verbatim and
  survives the switch back to the form, method switch resets, persisted tree
  restored, legacy flat entry dropped); existing method-pane tests moved to
  the form's #param-<field> ids and fill the required field before Run.
  Playwright smoke selector moved to #param-Text.
- Docs: design doc, Component-CentralUI.md entry-points bullet, CLAUDE.md.

CentralUI.Tests 1425/1425 green; Playwright project builds.

Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
2026-09-10 14:54:21 -04:00
Joseph Doherty 97f01ae98e fix(flow-debugger): final-review remediation — 2000-row trace pull, shared ExceptionHeader parser, surrogate-safe clip, wording, alarm source, register renumber (I1/I2/M1-M8) 2026-09-09 13:30:48 -04:00
Joseph Doherty 82739ec912 docs+rig(flow-debugger): step logs v1.4 specs, register rows, FlowLogDemo seed + verification step J 2026-09-09 12:56:39 -04:00
Joseph Doherty a071433096 docs: instance picker on Flow Debugger + Debug View
Note the shared InstancePicker combobox (2026-09-08 design) replacing
the flat instance <select> on both pages, in Component-CentralUI.md's
Flow Debugger and Debug View sections and the CLAUDE.md Flow Debugger
bullet.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-08 09:43:04 -04:00
Joseph Doherty 24026b25e2 docs: flow debugger — command counts read four commands / three run commands everywhere
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-08 01:22:49 -04:00
Joseph Doherty 4bb831fc41 docs: flow debugger script entry — component specs, CLAUDE.md, design pointers
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-08 01:21:43 -04:00
Joseph Doherty e15aa0e430 docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13)
The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):

- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
  execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
  fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
  capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
  client waits (F4), the 7-10 row estimate, the doubled central audit write per
  routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
  reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
  SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
  Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
  site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
  quoted: the method form, both CLI --trace-all-calls help strings, the CLI
  README, the ApiMethod entity doc, the manual migration script and the two
  Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
  the tracking-store test race fixed alongside them (86451382).

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 14:35:10 -04:00
Joseph Doherty 0ccbe140ae perf(audit-log): cached-call forwarder enqueues the audit row instead of awaiting its commit (194 ms -> 25 ms)
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.

TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.

Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.

Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 12:35:52 -04:00
Joseph Doherty d91135f960 perf(site-runtime): operation tracking store writes off the script thread (single-writer queue, batch per transaction)
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.

OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.

No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 12:25:58 -04:00
Joseph Doherty 7caf56f332 fix(central-ui): Flow Debugger outcome refresh merges rows by EventId instead of replacing the graph
Found in a browser Live run on the rig: the run drew API, DB and Notify nodes from
its inline rows; the 5 s pending-refresh re-read the tree through Replay before the
site's cached-call telemetry (12 s) had reached central, swapped the graph wholesale,
and the resolved DB node vanished (7 -> 6 steps) until a manual refresh. Polling had
already stopped because nothing was Pending.

MergeRefreshedGraph now unions the re-read rows into the run's known rows by EventId
(fresh copy wins), stores the union back on _raw and rebuilds through
FlowGraphBuilder, so a refresh only ever adds nodes. The union is exact: the inline
rows are the same AuditEvent objects the site writer persists (FlowTraceCollector).

Tests: new PendingRefresh_KeepsStepsTheLiveRunAlreadyHad_WhenReplayLagsBehindTelemetry
(fails on the old code). Two existing fixtures were internally inconsistent and only
passed under the old replace semantics: ParkedThenResolved built its Resolved outcome
with the Parked raw rows, and OutboundOutcome minted fresh event/correlation ids per
call, so no row could ever match across live and replay. Both now share rows per run.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 11:55:52 -04:00
Joseph Doherty 83ab2aee06 docs: Flow Debugger v1.2 — node Retry/Discard, ?correlationId= deep links, prediction deadlock finding; register row 111 closed
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 11:16:24 -04:00
Joseph Doherty 3c9f9da818 feat(flow-debugger): outbound calls on the External lane — CLI projection, specs, rig demo + verification step I
CLI flow-debug table now projects through FlowGraphBuilder (one row per step,
outbound lifecycles folded, new 'lifecycle' column; per-event fallback kept).
Specs: Component-CentralUI (four lanes, node anatomy, pending refresh),
Component-AuditLog (central method-script Database.* now audited; replay reads
the three outbound channels), Component-InboundAPI (DbOutbound row per helper
call), Component-ScriptAnalysis (REQ-SA-7 kinds + Async edges), design doc
v1.1 status line, CLAUDE.md bullet, deferred-work register row 111 (Retry/
Discard + ?correlationId= deep links deferred). Rig: seed-flow-outbound-demo.sh
(RigRest/RigDb/FlowDemo + OutboundDemo script + FlowOutboundDemo method/key)
and verify-delmia-notifier.sh step I (live rows inline, replay until terminal).
Rig verification result follows in its own commit.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 02:10:50 -04:00
Joseph Doherty 0a9ef87b54 docs: Flow Debugger — component specs, CLAUDE.md, CLI README, register row 110
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls
property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger
runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs;
CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the
PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel
retention example), Component-Communication (§4 trace flag + inline events with proto field
numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor),
CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console
capture), design doc status → implemented. CLI README sections landed in Tasks 3/16.
Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire
relationships only, unchanged by a new page — no edit.

Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
2026-09-06 00:38:41 -04:00
Joseph Doherty 59813da508 docs(claude): note index row length convention + sister-project-history split
scadaproj's umbrella index now keeps only short per-project summary rows,
parking long narrative in .claude/skills/sister-project-history/<project>.md
and shared-library CLAUDE.md files. Reflect that in the propagation rule so
future ScadaBridge index edits land in the right place.

Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
2026-09-03 10:48:04 -04:00
Joseph Doherty f47ceb55b1 docs: inbound API key store — shared SQL Server provider on central pairs (spec, runbook + central-pairs procedure, DBA DDL copy, topology, env inventory, CLAUDE.md, register row 108 closed with rig proof, row 109 Secrets backoff follow-up, Delmia spec §8)
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
2026-09-02 23:31:21 -04:00
Joseph Doherty 081e57f166 docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central kill/takeover 27 s, rejoin); register rows 107 (prod DelmiaRecv path) + 108 (per-node inbound API key store breaks central failover); CLAUDE.md bullet; plan + task file
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
2026-09-02 12:55:22 -04:00
Joseph Doherty 150c3f60d8 docs: tag-subscribe in-flight guard + backoff (Lever 1) — spec, CLAUDE.md, design-doc status 2026-08-28 14:53:23 -04:00
Joseph Doherty a9b23556eb feat(inbound-api): standby central node relays POST /api/* to the active peer instead of 503
A caller addressing a standby central node directly (or hitting Traefik inside its
/health/active probe window after a failover) used to get 503 STANDBY_NODE. The
standby now relays the request verbatim to the active peer over HTTP
(IStandbyRequestForwarder / HttpStandbyRequestForwarder) and replays the response;
method scripts and Route.To() still execute only on the active node.

- ScadaBridge:InboundApi:StandbyForwardPeerEndpoints (static list of the OTHER
  central node's HTTP base URL; empty = old 503 behaviour) + StandbyForwardTimeout
  (60s), both startup-validated.
- Peers tried in order: peer 503 stamped X-ScadaBridge-Standby, connection failure
  or timeout → next peer; all fail → 503 ACTIVE_NODE_UNREACHABLE. A 503 without
  the marker is the active node's own answer and is replayed.
- One-hop loop guard via X-ScadaBridge-Forwarded; 413 body cap now runs BEFORE
  the active-node check so a standby never buffers/relays an oversized body.
- AuditWriteMiddleware skips its own ApiInbound row for a relayed request so a
  call is audited once, on the node that executed it.
- Rig + env2 central nodes configured; spec (Component-InboundAPI.md → Standby
  forwarding), Component-Host.md options row, CLAUDE.md updated.
- 20 new tests (filter decision + relay mechanics + validator).

Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
2026-08-28 13:34:20 -04:00
Joseph Doherty afb27e7780 docs: CLAUDE.md — template picker on Create Instance (RevealKeys, ?templateId, Deployer-gated New Instance…) + the Single-mode SelectedKey/SelectedKeys trap
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 08:24:05 -04:00
Joseph Doherty a26edffd4c docs: create-time connection bindings (Central UI, DeploymentManager preview flatten, CLI cross-ref)
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 06:14:55 -04:00
Joseph Doherty 4388fa41e4 deps(localdb)+docs: consume LocalDb 0.3.4 — snapshot as_of_seq floors at last_acked_seq, ending the full-snapshot-on-every-reconnect loop (register row 106 closed, live-proven on the rig: second pair restart exchanged zero snapshots) 2026-08-20 10:00:37 -04:00
Joseph Doherty 25c15dd164 ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.

Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.

Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
2026-08-20 09:37:48 -04:00
Joseph Doherty 09989fdf32 deps(localdb)+docs: consume LocalDb 0.3.3 (FK-parent registration guard + atomic per-entry apply, arch-review P4-7); Low-wave integration docs — register rows 103 closed / 104 auto_vacuum legacy files / 105 FK-parent constraint; CLAUDE.md bulk-deploy up-to-date filter + --force, 0.3.3 line, infra/LDAP correction (P8-7); plan doc status 2026-08-20 08:56:50 -04:00