Commit Graph

118 Commits

Author SHA1 Message Date
Joseph Doherty c810bd0f7c feat(logging): bridge Akka logging to Serilog and copy traced runs' node-log lines into the Flow Debugger (ScadaBridge#120, #56)
#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
  so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
  stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
  name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
  structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
  To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
  which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
  ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.

#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
  launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
  executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
  ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
  low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
  ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
  registration is declared inside the launcher's and executor's try, so it ends before the catch
  blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
  its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
  scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.

Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.

Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 06:12:34 -05:00
Joseph Doherty 533414a16d fix(audit): maintenance-login roll-forward/backfill, archive orphan cleanup, head-export health (ScadaBridge#113 #114 #115)
#113: AuditLogPartitionMaintenance (monthly SPLIT RANGE) and the
backfill-source-node endpoint now use the AuditLog:MaintenanceConnectionString
login when set (unset = unchanged). Migration AuditMaintenanceGrants gives
scadabridge_audit_purger ALTER ANY DATASPACE and column-level UPDATE (SourceNode).
#114: archive cleanup deletes the writer's own *.tmp files and manifest-less
data files older than 24 h inside {Directory}/{YYYY} (DeletedOrphan, dry-run aware).
#115: head-export failures bump scadabridge.audit.chain_head_export.failures and
Degrade the /health/ready entry audit-chain-head-export (failing or stale > 2 intervals).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 23:47:47 -05:00
Joseph Doherty b95ee5a658 feat(audit): archive retention and superseded stale-archive cleanup (ScadaBridge#109)
Owner decisions recorded: the archive destination stays a directory/UNC
(no object-storage client; share + permissions named at enablement),
parquet-tools-only reading (no re-import), unsigned like the chain (#107).

AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed
AuditLog:RetentionDays, counted from the archived month's end) and
DeleteSupersededStaleArchives (default true) run after every purge tick.
A retried purge's stale Archived archive is deleted only when a later,
verifying, Purged archive of the same partition provably holds every row:
same prior anchor and the stale chain-end hash at its chain-end sequence
(row or tombstone), every unchained row present by EventId; otherwise it is
reported KeptStale. A sole copy left at Archived is never touched. Deletion
removes only the archive name's own three files in its folder, manifest
first, never a manifest-supplied path. `audit archive cleanup [--dry-run]`
= CleanupAuditArchives (Administrator).

AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy,
sole copy, retention and the path guard; each safety rule mutation-checked.
Follow-ups filed: #113, #114, #115.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 23:33:04 -05:00
Joseph Doherty c4d7ec9b96 feat(audit): off-box chain-head export, verify-heads, maintenance login (ScadaBridge#107)
Item 1 (owner decision: no signing). AuditLog:HashChain:HeadExport (default
off): both centrals periodically read every month's chain head (LastSeq,
LastHash, purge anchor, chained rows, tombstones; one snapshot under the
chain lock SHARED) and append the changed heads to a sink the DB login cannot
write - File (append-only JSONL per node per day, local or UNC) or Syslog
(RFC 5424 over UDP / TCP octet counting). `audit verify-heads --path` reads
those records back (file or syslog-collector lines) and the new
VerifyAuditChainHeads command checks each against the database: Matches /
Extended / Purged, or Mismatch / Regressed / MissingHead / MissingLink /
CountMismatch (exit 3). AuditChainHeadStoreTests forges a chain with the
production hasher: verify-chain says Intact, verify-heads says Mismatch.

Item 3. AuditLog:MaintenanceConnectionString (${secret:}-capable, unset =
today's single login) runs the chain job, chain verify / head export, the
partition switch-out + archive and the per-channel purge on
AuditMaintenanceContext (keyed audit-maintenance repository/archive, purge
actor prefers them). AuditMaintenanceLoginTests runs the hardened setup on
SQL Server: writer (datareader/datawriter/audit_writer) ingests and is
refused chaining and deleting; the chainer+purger login does the full cycle.
Two-login grants documented in Component-AuditLog.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 23:16:05 -05:00
Joseph Doherty 205a06a0f1 feat(transport): optional detached bundle signing with a pluggable key source (ScadaBridge#45)
Exports can now carry a detached ECDSA P-256 signature over the exact
manifest.json bytes in a new optional manifest.sig zip entry. The manifest
already pins the content hash, so a verified signature covers the whole
bundle, including the plaintext source environment the confirm step relies
on. Off by default (ScadaBridge:Transport:Signing): with the defaults,
exports are unsigned and imports accept signed and unsigned bundles.

- The bundle format does not change: bundleFormatVersion stays 1,
  manifest.json and the AES-GCM associated data are untouched, and older
  importers open entries by name and ignore manifest.sig. Three entries fit
  within the default MaxBundleEntryCount of 4.
- Import policy runs after the content-hash check and before any passphrase
  work. A signature that fails to verify against a trusted key is always
  refused. RequireSignedBundles additionally refuses unsigned bundles and
  bundles signed by an untrusted key. A refusal is BundleSignatureException
  (Commons), shown on the UI upload step and on CLI preview/import.
- The signature status, key id and signer are recorded on BundleSession and
  shown in the import wizard. They are also returned on PreviewBundleResult
  and recorded on the BundleImported audit row. BundleExported records
  Signed/SigningKeyId.
- Key material is pluggable. IBundleSigningKeySource and IBundleTrustStore
  are TryAdd'd; the defaults read Signing:PrivateKey (meant to be a
  ${secret:} reference) and Signing:TrustedKeys. Key ids are SPKI
  fingerprints. Startup validation rejects unparseable or non-P-256 keys.
- Compatibility proof: the checked-in fixtures pre-signing-plaintext and
  pre-signing-encrypted were exported by origin/main 3ff463b6, before this
  change, and are imported end to end with signing off and with signing
  configured.

The design leaves key-per-env vs per-org, trust distribution and rotation,
and the production default for RequireSignedBundles open. These are listed
in Component-Transport.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:42:28 -05:00
Joseph Doherty 7be4f19861 feat(audit): Parquet cold archive of expired AuditLog partitions before the purge (ScadaBridge#42)
Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.

- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
  management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
  UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
  tombstone file = the month's per-channel purge tombstones. Buffered row
  groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
  chain heads and tombstones in one transaction under the chain lock SHARED,
  writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
  .tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
  month's head and prior anchor) via tmp + rename, manifest last. Archives are
  never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
  overload. Under the EXCLUSIVE chain lock it re-counts the partition and
  re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
  both still equal the archive's, so the switch's checkpoint anchor is exactly
  the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
  takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
  publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
  from the archived rows and tombstones (AuditChainVerifier), and the chain end
  against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
  absolute; RowGroupSize, timeouts bounded.

No migration: the purger role already reads AuditLog, the head and tombstones.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 17:12:12 -05:00
Joseph Doherty fd4be80913 feat(audit): SHA-256 hash chain over the central AuditLog (ScadaBridge#41)
Audit roadmap T1 (alog.md §11.4): one chain per UTC calendar month of
OccurredAtUtc. Each chained row carries ChainSeq (gap-free within its month)
and RowHash = SHA-256(prev.RowHash || canonical(row)); the v1 canonical form
(AuditChainHasher) is pinned by a known-answer vector computed independently.

- Migration AddAuditLogHashChain: nullable ChainSeq/RowHash on dbo.AuditLog
  (metadata-only add), AuditLogChainHead (per-month chain end + partition-purge
  checkpoint anchor), AuditLogChainTombstone, two partition-aligned filtered
  indexes, scadabridge_audit_chainer role with a column-level UPDATE grant.
  Production script docs/plans/sql/AddAuditLogHashChain.sql.
- AuditLogHashChain chains unchained rows oldest month first under an
  exclusive sp_getapplock (zero timeout, so a dual-active central pair still
  yields one chain), with an optimistic head check and a ChainSeq IS NULL
  row-count check that roll a pass back instead of leaving a gap. The ingest
  path is untouched.
- Purges keep the chain verifiable: the partition switch records the month's
  checkpoint anchor and drops its tombstones; the per-channel DELETE writes a
  tombstone per chained row in the same transaction (OUTPUT INTO). Both, and
  the SourceNode backfill, hold the chain lock; the backfill skips chained
  rows. The switch staging table gains the two new columns.
- AuditChainVerifier reports the first break (partition, ChainSeq, EventId,
  kind) for edited, deleted, reordered rows and tail truncation.
- AuditLogHashChainService (central, default OFF via AuditLog:HashChain).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:33:11 -05:00
Joseph Doherty 063fb21815 fix(native-alarms): a 4-column CSV import keeps stored MES overrides on the CLI path too (ScadaBridge#104)
The bulk SetInstanceNativeAlarmSourceOverridesCommand handler always wrote the
entry's IsMesReportableOverride, so a CLI import of a legacy 4-column CSV (no
MesReportable column) reset every imported source's MES override to inherit. The
Central UI avoided it only by looking up the stored flag client-side.

NativeAlarmSourceOverrideEntry gains an additive trailing KeepStoredMesReportable
flag: when set, the server leaves an existing row's MES override untouched (a new
row inherits) and ignores IsMesReportableOverride; unset keeps the old meaning
(null = clear to inherit), so older senders are unaffected. The CLI and the Central
UI now build their entries through one rule (ToOverrideEntry on each row type):
no MesReportable column => KeepStoredMesReportable. CLI README and
Component-CentralUI updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 16:04:28 -05:00
Joseph Doherty 47f3a703bf perf(search): prefix-by-default Deployments and config-audit search; bounded Contains opt-in (ScadaBridge#50)
Deployments free-text search and the config-audit EntityName filter no longer
emit LIKE '%term%' by default. Prefix (LIKE 'term%') is the default; substring
is an explicit opt-in (DeploymentSearchMode.Contains / entityNameContains)
always bounded by a DeployedAt/Timestamp window (explicit, else 30 days).
Migration PrefixSearchIndexes adds DeployedBy, RevisionHash and EntityName
indexes. Deployments page gains a 'Match anywhere' checkbox.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:42:56 -05:00
Joseph Doherty 62fe1e685d feat(native-alarms): optional MesReportable column in the native-source override CSV import (ScadaBridge#102 item 4)
CLI import no longer has to reset MES overrides to inherit; a legacy 4-column file
still parses and the Central UI import keeps the stored flag for it.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:33:31 -05:00
Joseph Doherty 3dd492a77f feat(inbound-api): surface method compile errors to operators — api-method validate/recompile + compile state on get (ScadaBridge#75)
The data plane answers a broken inbound method with the generic "Script
compilation failed for this method"; the Roslyn diagnostic reached only the
central log, and the known-bad cache could not be observed or reset per method.

- InboundScriptExecutor keeps the diagnostics with each known-bad record and
  exposes GetCompileState(name, currentScript): IsCompiled / IsKnownBad /
  LastCompileError, judged against the current script text (a record for older
  text is reported as not compiled yet).
- Management: ValidateApiMethodCommand (compile the stored script, return
  ApiMethodCompileResult, register nothing, cache untouched) and
  RecompileApiMethodCommand (CompileAndRegister without re-sending the entity,
  drops the node's ApiMethodCache row, audited "Recompile"). Both Designer, like
  UpdateApiMethod; dispatched through the shared ApiMethodCompileOperations.
  GetApiMethod keeps the entity fields top-level and adds isCompiled,
  isKnownBad, lastCompileError and compileStateNode. Compile state is per
  central node and every result names the node.
- CLI: api-method validate --id, api-method recompile --id.
- Central UI: the API-method edit page shows the saved script's compile state
  and last error, with Validate and Recompile actions (same operations, same
  Designer gate, fresh DI scope).
- RequiredRoleMatrixTests entries; ManagementSurfaceParityTests covers the new
  CLI leaves; ApiMethodCompileDiagnosticsTests + an executor CompileState test.
- Docs: CLI README, Component-CLI/ManagementService/InboundAPI, components/*.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:30:46 -05:00
Joseph Doherty f5eed5e8fd feat(audit): persist OPC UA certificate-trust decisions to the central audit log (ScadaBridge#43)
New AuditChannel.Security with AuditKind.CertTrust / CertTrustRemove (appended, ordinals stable).
Both central entry points - the management actor handlers and the Central UI CertManagementService -
write a best-effort ICentralAuditWriter row after the site outcome is known (Delivered when all
reachable site nodes acked, Failed otherwise, incl. relay timeouts), naming actor, site, thumbprint
and connection. Shared builder: CertTrustAuditEvents.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:17:03 -05:00
Joseph Doherty f98369a786 feat(native-alarms): per-binding IsMesReportable on native alarm source bindings (ScadaBridge#59)
TemplateNativeAlarmSource.IsMesReportable + tri-state InstanceNativeAlarmSourceOverride
.IsMesReportableOverride flatten onto ResolvedNativeAlarmSource; NativeAlarmActor stamps every
mirrored condition and InstanceActor the quiet-binding placeholder, so ScriptAlarm,
Alarms.MesReportableAsync() and the #60 wire fields carry it. Additive to the 900-999 severity band.
Migration AddNativeAlarmSourceMesReportable. Management commands, CLI --mes-reportable, Central UI
(template dialog checkbox, instance override select), Transport DTOs/importer/diff and the revision
hash (true-only, so existing hashes are unchanged) carry it.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:09:54 -05:00
Joseph Doherty 6070cc2b9d perf(templates): resync and resolved-members read only the chain/subtree (ScadaBridge#89)
The last two whole-library template reads are gone. Both fed the
inheritance-only TemplateInheritanceResolver, which walks one template's
parent chain and never follows compositions.

- TemplateService.ResyncInheritedMembersAsync (operator repair, CLI + editor)
  now reads #86's GetTemplateReconcileClosureAsync (derived subtree + the
  template's chain, tracked). When nothing derives from the template the
  closure is empty, so it falls back to the new tracked
  GetTemplateChainAsync - resyncing a leaf reads only its own chain.
- ManagementActor.HandleGetResolvedTemplateMembers now reads
  GetTemplateChainAsync(id, trackChanges: false): the chain alone, untracked,
  with script bodies (the resolver projects script code, so #66's body-free
  analysis closure would not be equivalent).

The chain walk is factored out of TemplateReconcileClosure into
CollectChainAsync (one load per level, stops on parent cycles and dangling
parents) and shared by both.

Evidence:
- TemplateChainReadTests (#86 fixture): for every template plus an unknown
  id, the closure-backed resync issues exactly the writes, audit row and
  result of the whole-graph resync with GetAllTemplatesAsync wired to throw;
  the resolver answers identically from the chain and the whole graph.
  17 of these fail against the old service; 13 fail with the leaf chain
  fallback removed (mutation-checked).
- TemplateChainRepositoryTests (SQLite, recorded SQL): a 4-deep chain loads
  4 templates / 8 attributes instead of 46 / 92, one split-query set per
  level, untracked on request and tracked for the resync.
- ResolvedTemplateMembersReadTests: the actor answers from the untracked
  chain read and never calls GetAllTemplatesAsync (fails on the old actor).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:30:26 -05:00
Joseph Doherty c29b99c63b perf(templates): descendant reconcile loads only the subtree + chain (ScadaBridge#86)
TemplateService.ReconcileDescendantsAsync - run after every base member
add/update/delete, template and native-alarm-source alike - loaded the whole
tracked template graph (every template, member and script body) on each call,
which after #66 was the dominant cost of those paths. It now calls the new
ITemplateEngineRepository.GetTemplateReconcileClosureAsync: the same tracked,
fully-included shape as GetAllTemplatesAsync, restricted to the edited
template's derived subtree plus its own inheritance chain. The reconcile is
inheritance-only (TemplateInheritanceResolver never follows compositions) and
every descendant's chain runs through the edited template, so that set is
everything it reads.

The walk (Commons TemplateReconcileClosure, the descendant-direction twin of
#66's TemplateAnalysisClosure) loads one children batch per subtree depth and
one batch per ancestor level, terminates on parent cycles and stops at
dangling parents. A template with no children - the common case - costs one
empty probe and loads nothing.

Evidence:
- TemplateReconcileClosureTests: for every template of a graph with a deep
  subtree, an ancestor above the edited template, a composed module,
  look-alike unrelated trees, a parent cycle and a dangling parent, the
  closure-backed reconcile issues exactly the writes (add/update/delete,
  audit, save) of the old whole-graph reconcile, and the resolver answers
  identically for every descendant. Fails with the old service (whole-graph
  load wired to throw) and with the ancestor walk removed (mutation-checked).
- TemplateReconcileClosureRepositoryTests (SQLite, recorded SQL): 5 templates
  / 10 attributes instead of 47 / 94, tracked with script bodies; a leaf costs
  one probe query and tracks nothing.
- TemplateServiceTests: attribute add + update propagate two levels down with
  GetAllTemplatesAsync wired to throw; existing propagation tests now run
  against the closure via ServeReconcileClosureFromFullGraph.

ResyncInheritedMembersAsync (operator action) still reads the whole graph.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:10:31 -05:00
Joseph Doherty 0eac956d25 feat(health): report permanently rejected site audit rows on the site health report (ScadaBridge#69)
SiteAuditBacklogSnapshot gains an optional trailing RejectedCount (gauge of
Rejected forward-state rows, read from the site store), carried on the wire as
SiteAuditBacklogSnapshotDto.rejected_count = 4 (additive; old sites read as 0,
old central ignores it). Health page shows an 'Audit rows rejected' badge.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:05:48 -05:00
Joseph Doherty 792f16c2d5 perf(templates): member add/update loads only the chain + composition closure (ScadaBridge#66)
Attribute/alarm/script add (collision check) and update (parent-lock lookup)
used GetAllTemplatesForAnalysisAsync, materialising every template, attribute,
alarm and script in the library per edit. They now call the new
ITemplateEngineRepository.GetTemplateAnalysisClosureAsync: the same untracked,
body-free analysis projection, restricted to the template's inheritance chain
plus the composition closure of every chain member - exactly what
TemplateResolver and CollisionDetector walk (a composed module's own parent
chain is never climbed, so it is not loaded). Adds use the edited template's
closure, updates the parent's.

The walk (Commons TemplateAnalysisClosure) loads one batch per closure depth,
requests each id once, terminates on cycles and skips dangling edges.

Evidence:
- TemplateAnalysisClosureTests: for every template of a graph with chains,
  nested + diamond compositions, a derived module, cycles, dangling edges and
  colliding unrelated templates, ResolveAllMembers / DetectCollisions /
  FindMemberByCanonicalName answer identically from the closure and the whole
  graph (mutation-checked).
- TemplateAnalysisClosureRepositoryTests (SQLite, recorded SQL): 6 templates /
  12 attributes loaded instead of 47 / 94, in 4 queries (one per depth),
  untracked, script bodies never selected.
- TemplateServiceTests: two new tests with the whole-graph load wired to throw
  (fail on the old service); every existing add/update test now runs against
  the closure via AnalysisClosureMock.

ReconcileDescendantsAsync still loads the tracked full graph; not in scope.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 12:52:29 -05:00
Joseph Doherty 93fbd2e030 fix(transport): pin a compressed copy of the decrypted bundle on the import session, release the upload buffer early, document the footprint (review 2026-10-21 04/M2) 2026-10-22 09:38:16 -04:00
Joseph Doherty 6bcca4b344 feat(templates): GetTemplateStubsAsync untracked identity/edge projection 2026-10-22 06:06:27 -04:00
Joseph Doherty 0ffd4bdbea docs(src): fix CommentChecker findings across src/ — 1,032 → 5 (all false positives)
Comment/XML-doc changes only, no code logic:
- strip project-tracking references (WP*, P*-*, row-NN, #NN, Task N) from
  comments, keeping the explanatory rationale
- add missing summaries, <param>, <returns>, <typeparam> and <inheritdoc/>
- replace misused <inheritdoc/> with real docs; where <inheritdoc/> was
  combined with useful remarks, move the rationale into // comments
- drop stale replicationActor <param> on DeploymentManagerActor ctor

Remaining 5 are false positives (CWE-1236 x2, failures-1, year-0001, M365).
Build 0 errors / 0 warnings.
2026-10-07 14:12:25 -04:00
Joseph Doherty 67a3950f74 fix(site-secrets): one shared secret: reference parser for scanner and MxGateway adapter
Final review M3: SecretReferenceScanner.TryParseSecretReference (trimmed, case-insensitive
prefix, trimmed name, empty = not a reference, ${secret:} never matches) is used by the scanner
and by MxGatewayDataConnection's ApiKey resolution, so a padded or case-variant reference
resolves the name central ships. A bare 'secret:' ApiKey now fails closed. M7: unit test pins
stamped hash == ComputeCurrentArtifactHashesAsync with a non-empty Secrets list.
2026-09-28 21:08:34 -04:00
Joseph Doherty 01cb1f5652 feat(secrets): SecretReferenceScanner derives a site's secret refs from its data-connection configs 2026-09-28 17:27:28 -04:00
Joseph Doherty b3021e7c5f feat(scripts): read-only, allow-listed Files global for site scripts; Delmia test receiver parses recipe files
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.

Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
2026-09-28 10:36:42 -04:00
Joseph Doherty 17d9a4cd2c feat(scripts): ScriptAlarm.IsMesReportable + Alarms.MesReportableAsync() mirrored on both compile surfaces
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
2026-09-11 10:18:05 -04:00
Joseph Doherty b07394447f feat(commons): IsMesReportable on TemplateAlarm, ResolvedAlarm; IsMesReportableOverride on InstanceAlarmOverride
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
2026-09-11 09:52:34 -04:00
Joseph Doherty e5c536c6b7 feat(commons): HandshakePhase + HandshakeResult for the reset-aware handshake (#36)
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
2026-09-10 15:18:10 -04:00
Joseph Doherty 9ee402bbcd Resolve all 46 low-severity architecture review findings 2026-09-09 22:56:36 -04:00
Joseph Doherty 9a87d7a328 Resolve all 33 medium-severity architecture review findings
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.

Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.

Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.

Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.

Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.

Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.

Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
2026-09-09 22:05:30 -04:00
Joseph Doherty 146a13238f Resolve all high-severity architecture review findings 2026-09-09 16:13:58 -04:00
Joseph Doherty 97f01ae98e fix(flow-debugger): final-review remediation — 2000-row trace pull, shared ExceptionHeader parser, surrogate-safe clip, wording, alarm source, register renumber (I1/I2/M1-M8) 2026-09-09 13:30:48 -04:00
Joseph Doherty 16cdf0adf1 Merge wt/step-logs-t3: Task 3 FlowGraphBuilder log-line attachment 2026-09-09 11:40:05 -04:00
Joseph Doherty 395f38e019 feat(commons): FlowGraphBuilder folds ScriptLog rows onto their owning step (D2) 2026-09-09 11:33:54 -04:00
Joseph Doherty a45c0d47c5 Merge wt/step-logs-t2: Task 2 FlowTraceCollector low-priority lane 2026-09-09 11:33:15 -04:00
Joseph Doherty b73962831c refactor(commons): FlowTraceCollector.Add(lowPriority:) instead of a fourth public method (Commons method-count constraint) 2026-09-09 11:28:01 -04:00
Joseph Doherty befb0e3805 feat(commons): FlowTraceCollector low-priority lane for log lines (D7) 2026-09-09 11:26:05 -04:00
Joseph Doherty 8f88cab7f8 feat(commons): ScriptLog audit kind, log-line trace extra, FlowStep.ErrorDetail/Logs (flow-debugger step logs D1/D3) 2026-09-09 11:25:55 -04:00
Joseph Doherty 859922ba17 fix(central-ui): Flow Debugger spawn edges — real arrowhead + bolt badge; spawner picked by trigger attribute
The trigger-cascade (Spawn) edge used a lightning-bolt marker IN PLACE of the arrowhead. On the upward
Tags→Site hop the bolt sat on top of the riser and read as a broken arrow (rig screenshot 2026-09-08).
Every connector now ends in an ordinary arrowhead; the bolt is a badge (circle + bolt) centred on the
connector's vertical run (FlowGraphView.SpawnBadgePoint, kept in step with OrthogonalPath/AsyncPath).

Same screenshot, second defect: the cascade spawned by "RecipeDownloadFlag = false" was drawn from the
LAST write of the burst (RecipeProcessedFlag) because all four writes share a millisecond and the
spawner rule was time-only. FlowGraphBuilder now prefers the parent's last AttributeWrite of the attribute
the child's triggerAttribute names (at or before the child, else the last such write); the time/position
rules remain the fallback when the child names no attribute or the parent never wrote it.

Verified live on the rebuilt rig (DelmiaSim01 / DelmiaReceiverSimulator): both spawn edges show an
arrowhead + badge, the second one originates from the RecipeDownloadFlag write. bUnit + builder tests
updated/added; Playwright FlowDebuggerTests 4/4.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-08 09:04:39 -04:00
Joseph Doherty 952b41f0f7 feat(commons): ScriptTriggerAttribute — monitored attribute of a ValueChange/Conditional trigger (flow debugger script entry)
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-08 01:07:38 -04:00
Joseph Doherty f17bae093a fix(flow-debugger): review follow-ups — redact runner-synthesized rows, spawner rule, budget doc, CLI ceiling note (arch-review remediation)
Code review of the merged arch-review remediation (Tasks 1-13) found four items.

1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
   inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
   (whose writer redacts a private copy) and then put RAW into the returned Events —
   so a Live run's caller saw the unfiltered row while Replay of the same execution
   showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
   methods now resolve IAuditRedactor once per run and place the filtered copy into
   the result (including the early-return path on a failed attribute write); the raw
   row still reaches the writer, which redacts itself.

2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
   Since executions are ordered as contiguous blocks, position[p] < childPosition
   holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
   Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
   rule, in order: (a) the last non-async, non-Response parent step occurring at or
   before the child's first step — parent and child of a trigger cascade share the
   site clock, so this picks the write that actually fired it; (b) the position rule
   as fallback, which is the mixed-clock case F5 was written for (a child clock
   behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
   The class remark's stale "Replay order is unchanged" claim is replaced with what
   actually happens to a central Database.* call made after Route.To().Call.

3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
   "well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
   UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
   framing — and raise it only against a real serialized-size measurement.

4. CLI --timeout-seconds help + README row now say the management endpoint caps
   flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
   5 min), so a larger client value still 504s at the server's ceiling.

Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.

Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 14:44:34 -04:00
Joseph Doherty 1dfb9cee20 fix(commons): Flow graph orders executions by emission sequence, timestamps only within one clock (arch-review F5)
Central rows carry central's clock, site rows the site's, so the global
OccurredAtUtc sort let a 1-2 s skew float a whole site execution ahead of the
inbound row / RouteCall that started it, and pushed the trigger-cascade spawner
search onto its parentChain[0] fallback.

FlowGraphBuilder.Build now orders executions by the input index of their
earliest-given row and orders steps by timestamp only INSIDE one execution
(one clock), breaking ties by kind rank then input index. A folded outbound
node keeps its earliest-by-time row's identity, which need not be the row given
first, so its index is the minimum over the rows folded into it. The spawner
search uses position in the ordered list rather than a raw timestamp compare.
FlowDebugRunner.RunMethodAsync keeps the collector's emission order instead of
re-sorting; the value-change path keeps its OrderBy (all rows share one site
clock). Replay input arrives timestamp-sorted from the repository, so its order
is unchanged.

The DelmiaShaped test fixture is still shuffled within each execution (the
inside-one-execution timestamp rule is unchanged) but now lists the executions
in emission order, since that is what fixes their relative order.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 14:28:12 -04:00
Joseph Doherty e79452d2de fix(site-runtime): Flow Debugger inline trace copy is redacted and byte-capped at the collector (arch-review F1, site half)
Every emit site hands FlowTraceCollector the same AuditEvent it hands the
writer, but redaction and payload capping lived inside the writers, which
return a NEW filtered record — so the inline copy that rides
ScriptCallResult.TraceEvents / RouteToCallResponse.TraceEvents was the RAW
event: no per-target SQL-parameter redaction, no header scrubbing, no 8 KB /
64 KB truncation. Live and Replay of the same execution could therefore show
different payloads, and the 500-event-only cap left the reply free to breach
the 4 MB gRPC receive limit — which would fail a real call, since the same
reply carries the script's return value.

The collector now applies the host IAuditRedactor itself (outside the lock;
the writers keep their own idempotent Apply) and enforces a byte budget
(DefaultMaxBytes = 1 MB over the redacted DetailsJson) alongside the event
cap, flagging Truncated when either trips. It moves to Commons
(Types/Debugging) so the central FlowDebugRunner can share it.

The redactor is a singleton, so ScriptRunLauncher resolves it from the ROOT
provider and keeps constructing the collector before the try — the traced
ScriptRun row is emitted from the catch blocks too, including the
shed-at-dequeue path that throws before the DI scope exists.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 14:15:38 -04:00
Joseph Doherty 95773aa9fd feat(site-call-audit): additive TrackedOperationId query filter for ?correlationId= deep links
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 11:00:03 -04:00
Joseph Doherty 5497ffd181 feat(notification-outbox): additive NotificationId query filter for ?correlationId= deep links
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 10:59:49 -04:00
Joseph Doherty 7e3e809a70 feat(flow-debugger): FlowStep carries the audit CorrelationId; CLI table adds a correlation column
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 10:58:50 -04:00
Joseph Doherty 865c4be651 fix(commons): FlowGraphBuilder review fixes — leading async steps link from the spawner, robust async rule, lifecycle tie-break, total DbSubtitle
Review of Task 2 (2026-09-06 outbound-calls plan): (1) an outbound step that
precedes the first chain step of its execution — the default Replay shape when
a run's ScriptTrace rows are absent — was orphaned or given a solid spawner
edge; leading async steps now hang off the spawner on Async edges and the
first non-async step gets the normal link. (2) IsAsync keyed on the first row
being CachedSubmit; it now mirrors the projector's sync rule (first row not
ApiCall/DbWrite) so a group missing its submit row still branches. (3) Same-
tick rows tie-broke by GUID; new LifecycleRank (submit < attempt < resolve,
non-terminal < terminal) makes first/latest attribution order-independent.
(4) DbSubtitle used GetInt32 (FormatException on out-of-range/fractional) —
now TryGetInt32, never throws. Also: External-lane edges never count as
central↔site crossings; DurationMs clamped; FlowStep doc updated.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 01:49:40 -04:00
Joseph Doherty 29129ef17a feat(commons): FlowGraphBuilder folds outbound lifecycles into one node; async/spawn edges; Response from the inbound execution
Fold rule: outbound rows sharing (kind, CorrelationId) become one FlowStep with
a Lifecycle list — first row's identity/time/input, latest row's outcome/
subtitle/output/error, DurationMs = sync row's own or submit→terminal wall time
(null while pending). Chain skips async steps (Notify, or first row
CachedSubmit) and links them with Async edges from the preceding chain step.
Spawner edges from a non-RouteCall are Spawn: true. The Response links from the
last non-async step of the inbound execution (the RouteCall), not the globally
last cascaded step (D9). FlowGraphView places a Response at max column + 1 so
the D9 origin does not pull it left (pulled forward from Task 7 to keep the
layout test green).

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 01:42:48 -04:00
Joseph Doherty e26616bebc feat(commons): project outbound API/DB/notification rows onto the External lane
ResolveKind maps ApiOutbound/DbOutbound/Notification to ApiCall/DbCall/Notify;
ResolveLane puts the three on FlowLane.External; outbound outcomes map
Delivered→Ok, Failed→Failed, Parked/Discarded→Parked, else Pending; subtitles
'{status} · sync|cached', '{op} · n rows' from a DbWrite Extra, '{status}' for
Notify. New FlowStepProjector.IsOutbound helper.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 01:38:41 -04:00
Joseph Doherty cc1bbf3975 feat(commons): Flow Debugger outbound kinds, External lane, Pending/Parked outcomes, FlowLifecycleRow
FlowLane.External (3); FlowStepKind.External replaced by ApiCall/DbCall/Notify;
FlowStepOutcome gains Pending/Parked; new FlowLifecycleRow + additive
FlowStep.Lifecycle; FlowEdge gains Async/Spawn (default false). Predictor and
graph-view references renamed compile-only (full work in Tasks 6/7).

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
2026-09-06 01:37:29 -04:00
Joseph Doherty ec7b5b2ea3 feat(commons): FlowStep/FlowGraph projection + FlowGraphBuilder for the Flow Debugger
Central-side projection of ScriptTrace/ApiInbound audit events (design §7.2): lanes
Central/Site/Tags, in-execution chaining, spawner→child edges (RouteCall by
childExecutionId), inferred write→satisfied-wait echo edge, synthesized Response node.
ScriptTraceExtra is the camelCase Extra payload shared by every emitter.

Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
2026-09-05 23:41:50 -04:00
Joseph Doherty 352e0bfa37 feat(audit): ScriptTrace channel + RouteCall/ScriptRun/AttributeWrite/AttributeWait kinds
Flow Debugger (design 2026-09-05 §5.1): trace steps are audit rows. New
AuditChannel.ScriptTrace appended after Cluster and four kinds appended after
ManualFailover so stored ordinals never move; lock-in tests + alog.md §4 updated.

Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
2026-09-05 23:41:50 -04:00