Commit Graph

588 Commits

Author SHA1 Message Date
Joseph Doherty d15c5f02ea perf(dcl): batch subscribe/read/write seam, bounded reconnect, sharded subscriptions 2026-08-14 21:14:04 -04:00
Joseph Doherty d4bb4e17f2 docs(plans): AddTemplateScript OOM re-tested on wonder — no longer reproduces (both forms, throwaway template, <1s) 2026-08-12 18:28:05 -04:00
Joseph Doherty d6b1942932 docs(plans): record uptime-matched wonder working-set re-read — 4.5x/2.0x reduction confirmed flat 2026-08-12 18:25:00 -04:00
Joseph Doherty 4a026d256e docs(plans): record wonder-app-vd03 live-gate PASS — leak closed in production (8.3x metadata collapse) 2026-08-12 18:10:02 -04:00
Joseph Doherty 20f2ae05ca docs(plans): record rig live-gate PASS for the resolver cache (+2 objs/compile vs +289 pre-fix) 2026-08-12 17:49:29 -04:00
Joseph Doherty df17b53f08 docs(plans): mark resolver-cache tasks 1-8 complete 2026-08-12 16:50:43 -04:00
Joseph Doherty d412fc3696 docs(scripts): record the shared caching metadata resolver invariant 2026-08-12 16:44:44 -04:00
Joseph Doherty 901cec9026 docs(plans): script-compile metadata resolver cache plan 2026-08-12 16:30:53 -04:00
Joseph Doherty 5a781c706c fix(scripts): build Roslyn ScriptOptions once per process, not per compile
ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.

Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.

Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
  compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run

ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.

Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.

This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
2026-08-12 09:51:08 -04:00
Joseph Doherty 006202f3c7 fix(audit): fail closed when a configured redactor is unavailable (#35)
Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.

AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.

Two changes:

1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
   the whole payload and increments AuditRedactionFailure, reusing the existing
   safety-net path. "Not configured at all" stays permissive — conflating those
   two states is the actual defect, so both are pinned by tests.

2. Precompile off the hot path. The audit-log roadmap specifies patterns are
   "precompiled at startup; rejected if compile takes >100ms"; the implementation
   had drifted to compiling lazily on first event, which put a wall-clock budget
   on a hot path under production load. RegexOptions.Compiled emits IL during
   construction, so a busy node could blow the budget on a perfectly valid
   pattern. Warm-up now runs at construction and on every options reload. The
   residual window between a reload and its warm-up is safe because that path
   now fails closed.

Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).

Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.

Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.

Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
2026-08-12 03:04:50 -04:00
Joseph Doherty fdc6b0c2bb chore(secrets): adopt ZB.MOM.WW.Secrets 0.6.2 and close the pre-host guard gap
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.

The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.

The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.

Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
2026-08-11 09:16:31 -04:00
Joseph Doherty d14e0ee4b1 fix(ui): gate detail modals on user intent, not on the row resolving
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).

The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.

Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.

Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.

The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
2026-08-11 06:03:06 -04:00
Joseph Doherty 9e243493fb ui: Central UI density/consistency sweep + Theme 0.4.1
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.

The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.

Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.

NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.

Build 0/0; suite green apart from that one intermittent failure.
2026-08-11 05:50:12 -04:00
Joseph Doherty b6f383a225 docs(notifications): record EWS live-gate PASS (rig -> on-prem Exchange, Delivered first attempt) 2026-08-10 07:29:06 -04:00
Joseph Doherty ba994a59c5 docs: final-review nits — test-stub wording, README tech-stack row, UI-audit follow-up 2026-08-10 07:07:00 -04:00
Joseph Doherty 05d0631cdd chore(plans): mark EWS transport plan tasks 10-11 complete 2026-08-10 07:00:12 -04:00
Joseph Doherty 00d8a923af docs(notifications): EWS transport docs; close Q12 as superseded; design-doc corrections from execution reviews 2026-08-10 06:56:56 -04:00
Joseph Doherty 9f4d7d4bcb docs(notifications): EWS email transport implementation plan (11 tasks) 2026-08-10 06:05:04 -04:00
Joseph Doherty b2ea9c6c74 docs(notifications): design EWS email transport for the outbox; close Q12 as superseded
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
2026-08-10 05:56:41 -04:00
Joseph Doherty e697477c1f feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 11:39:28 -04:00
Joseph Doherty 4d7f09d550 test(secrets): central-shared-store live gate 5/5 — shared SQL store + hub failover proven on the rig
Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 11:28:58 -04:00
Joseph Doherty f6c3f7c593 test(secrets): live gate 4/4 — check 4 re-run and PASSES on 0.4.1
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.

The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.

Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.

Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.

The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.

Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.

4/4. Merging.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 08:51:39 -04:00
Joseph Doherty 9d5cf7100e test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 08:01:55 -04:00
Joseph Doherty 4df3a55824 docs: truth sweep — retire stale registers, reconcile ledgers with shipped state
- deferred.md: DELETED (git rm) — stale 2026-07-10 duplicate of the canonical
  deferred-work register; this completes archreview R2-08 T11 (the file was
  tracked, not untracked as the task assumed)
- ScadaBridge-docs-issues.md, ScadaBridge-docs-fixed.md: DELETED (git rm) —
  generated 2026-07-10 CommentChecker reports, already consumed; completes
  R2-08 T13 (also tracked, not untracked)
- stillpending.md: prepended historical-snapshot banner (2026-06-15 audit;
  Tier-1 table is not current open work)
- docs/plans/phase-8-checklist.md: replaced the unevidenced 'Complete / All
  passing' stub with the honest state per register row 25 (WP-4 target-scale
  load test never run)
- archreview/plans/00-MASTER-TRACKER.md: R2-01 T2 live failover drill
  annotated RESOLVED 2026-08-01 (PLAN-R2-01 T4 + docker/failover-drill.sh +
  SbrFailoverTests); R2-08 T11/T13 recorded completed by this sweep
- docs/plans/2026-07-22-clusterclient-to-grpc-plan.md: P3 deferred-RPCs note
  updated (all four live-proven 2026-08-01, 1c99d6fa); ClusterClientSiteAuditClient
  naming follow-up marked DONE (63c16d69)
- docs/plans/2026-05-28-opcua-tag-browser.md.tasks.json: Task 19 flipped to
  completed (manual smoke PASS 2026-08-01, 6dc5d94c)
- archreview/plans/PLAN-R2-0[1-8]*.tasks.json: all-pending manifests reconciled
  with the authoritative tracker (round 2 merged @ 1930f19b) — flipped to
  completed except R2-08 T1/T2 which remain pending needs-user
- docs/operations/2026-07-16-secrets-clustered-master-key.md: correction banner
  (SQL-hub replication shipped 8e12f994; KEK-rotation + clustered-secrets
  runbooks ship with ZB.MOM.WW.Secrets)
- docs/plans/2026-07-19-localdb-phase2-live-gate.md: external-system-delete
  observation annotated RESOLVED (2d03f2d5 reconciles deletions incl.
  external_systems)

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 01:56:33 -04:00
Joseph Doherty b8f91bab2d feat(cluster): enable the #33 bootstrap guard on the docker rig — live gate PASS
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
2026-08-02 01:04:55 -04:00
Joseph Doherty 3c9b101dfc docs(alarms): document the Alarms accessor + AckTime, tick MES plan Phase 1
Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:

  - Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
    it is not scope-prefixed, why it is read-only, why placeholder rows are
    included), the full ScriptAlarm shape, AckTime on the enriched
    AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
    persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
    builds its CDC triggers from the column list at registration time).
  - Component-DataConnectionLayer.md already carried the AckTime section in the
    first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
  - Component-ScriptAnalysis.md: accessors returning domain types return the
    SAME type on both surfaces, and the trust-model note that a deny-list needs
    no entry for a new globals member -- only that its return type resolves in
    a permitted namespace.
  - Component-InboundAPI.md records the NEGATIVE decision: there is
    deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
    per-instance and lives on the site's Instance Actor, so the read goes
    through a routed site script and the filtering happens where the data is;
    central stays a thin router.
  - CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
  - The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
    what was actually built (incl. the two choices that differ from the plan's
    "or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
    and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
    stay open -- they are deployed config and need a live rig.
2026-08-01 13:12:46 -04:00
Joseph Doherty 01bcca992c feat(alarms): thread an additive AckTime through the native-alarm mirror
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.

Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.

Provenance is honest, never fabricated:
  - OPC UA A&C supplies a TRUE ack instant, so we now select it:
    AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
    index 18, APPENDED so the positional reads at 0-17 keep their meaning.
    Servers that omit the field fall back to the event's own Time.
  - MxAccess Gateway supplies none, so the ack transition's own timestamp is
    used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
    re-subscribe snapshot restores one from LastTransitionTimestamp rather
    than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.

Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.

Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.

Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
2026-08-01 13:12:04 -04:00
Joseph Doherty 6dc5d94cb9 docs(plans): tag-browser Task 19 manual smoke PASS 2026-08-01 — online + offline paths verified live 2026-08-01 12:41:24 -04:00
Joseph Doherty 819ca4d7ce docs(plans): retire the folder-hierarchy manual smoke — drag-drop steps obsolete under the [PERM] decision, rest covered by bUnit suites + CLI parity 2026-08-01 12:37:48 -04:00
Joseph Doherty a949fe6f57 docs(plans): retire the env2 + transport manual checklists — core scenario run live 2026-08-01 (caught + fixed the create-missing FK bug), rest covered by automated suites + #31 2026-08-01 12:34:34 -04:00
Joseph Doherty 1c99d6fa8d docs(grpc-plan): live gate fully closed — ExecuteOpcUa/ExecuteRoute/parked-retry/TriggerSiteFailover all live-proven 2026-08-01 2026-08-01 12:25:27 -04:00
Joseph Doherty 9ab27d5d61 docs(register): rows 27-28 — ES retry config never reaches sites (found live), failover dialog mislabeled Delete 2026-08-01 12:02:23 -04:00
Joseph Doherty f6822f8f45 docs(m10): close follow-up #163 — InstanceConfigureListOverrideTests verified green (#207 re-verified too) 2026-08-01 11:28:46 -04:00
Joseph Doherty 410349767d style(centralui): full-app bg-light/bg-white -> theme-aware utility sweep (M10 residual)
T34c only fixed the bounded modal-surface offenders. Bootstrap 5.3's bg-light
and bg-white are fixed light values that do NOT flip under [data-bs-theme=dark],
so every remaining use was a dark-mode contrast break. 35 swaps across 19 files:

  surface / <pre> / <code>  bg-light                -> bg-body-secondary
  panel                     bg-white                -> bg-body
  neutral badge             bg-light text-dark      -> bg-secondary-subtle text-secondary-emphasis
  muted badge / input group bg-light text-muted     -> bg-body-secondary text-body-secondary

DELIBERATELY LEFT (7 sites): the neutral member of a status-badge switch or
ternary whose siblings are all solid, non-theme-aware colours (bg-success,
bg-danger, bg-warning) — swapping only the neutral one to a subtle token breaks
the visual weight of the set, so these stay until the whole family is retoned:
  Topology.razor:513 (Current badge) and :588 (InstanceState.NotDeployed)
  InstanceConfigure.razor:1520 (same InstanceState switch)
  NotificationReport.razor StatusBadgeClass fallback
  TransportImport.razor ConflictKind badge fallback
  SecuredWrites.razor text-bg-light fallback (a text-bg-* family with no
    theme-aware member at all)
  Health.razor:377 depth ternary
Also untouched by design: SchemaBuilder.razor:88 (already bg-light-subtle,
theme-aware), bg-dark text-light console panels, and site.css / #reconnect-modal.

InstanceConfigure.razor is edited concurrently elsewhere; its change here is 5
pure class-string swaps on existing lines, no reflow. No test asserted any of
these classes. Also marks all four M10 residuals done in the plan doc.
2026-08-01 11:28:20 -04:00
Joseph Doherty a506b19d17 refactor(centralui): migrate TemplateEdit's four page-embedded modals to the DialogService host (M10 residual)
TemplateEdit was the last page still hand-rolling its own modal chrome after
M10 — T34c only tokenized its backdrops. All four member-authoring forms
(Attribute, Alarm, Native Alarm Source, Script) now open through
IDialogService.ShowAsync, so the single DialogHost in MainLayout owns the
backdrop, focus trap, Escape and focus restoration.

Pattern copied from the already-migrated pages (MoveDataConnectionDialog +
Templates' Move/Rename dialogs): each body is its own component beside the page
taking a DialogContext<bool>, owning its form state, rendering validation and
server errors INLINE while staying open, and closing with Close(true) only once
the save succeeded — at which point the page reloads. An inline RenderFragment
would NOT have worked: DialogHost renders the captured fragment in its own tree,
so the page's StateHasChanged could never refresh it.

Persistence deliberately stayed on the page (it owns TemplateService, the
inherited-member rules, and the repository-direct native-source path) and is
reached through an OnSaveAsync delegate returning null on success or the message
to display. Behaviour preserved verbatim, including the List-attribute encode +
Decode round-trip check, the name/trigger-type read-only-on-edit rules, the
duplicate-native-source-name guard, and NormalizeExecutionTimeout.

TemplateScriptDialog keeps all four tab panels mounted (Monaco and the JSONJoy
island must not tear down on tab switch) and hosts the Test Run panel, which
needs the live unsaved editor buffer, so it injects ScriptAnalysisService
directly and cancels an in-flight run on dispose.

Extraction moved markup that three structural source-scanning tests pinned;
all three were repointed at the new files rather than weakened:
  - TemplateNativeAlarmSourceEditorTests (+ a new test asserting the form is
    host-mounted and the body renders no chrome of its own)
  - AttributeListEditorTests (list-editor reveal now in the dialog body)
  - TestRunWarningTests (Real I/O warning travelled with the script panel)

Build 0/0; CentralUI.Tests 973/973 green.
2026-08-01 11:28:20 -04:00
Joseph Doherty fdfd5e1b27 feat(centralui): TreeView full WAI-ARIA keyboard navigation (M10 residual R7)
TreeView<TItem> handled only Enter/Space on the chevron; the tree itself was
unreachable by keyboard. Implements the WAI-ARIA tree pattern:

- Roving tabindex on li[role=treeitem] — exactly one node is tabbable, so the
  whole tree is a single Tab stop. Target resolved per render: last-focused node
  while still visible -> SelectedKey -> first visible node. Browser focus follows
  via a per-node ElementReference + FocusAsync in OnAfterRenderAsync, guarded by
  the same JSException/JSDisconnectedException/InvalidOperationException triple
  the context-menu focus already used (a no-op under bUnit).
- ArrowDown/ArrowUp move between VISIBLE nodes; ArrowRight expands a collapsed
  branch else moves to first child; ArrowLeft collapses an expanded branch else
  moves to parent; Home/End jump to first/last; Enter/Space activate through the
  SAME path a click takes (OnContentClick / OnCheckboxToggle) so selection
  semantics never diverge between input modes.
- ARIA: aria-level, aria-posinset, aria-setsize added; aria-selected now renders
  true/false on selectable+checkbox trees and stays absent on non-selectable ones.
- Event scoping: @onkeydown:stopPropagation on the li, chevron, checkbox and
  content slot, so nested nodes do not double-handle and consumer controls inside
  NodeContent keep their own key handling. Browser scroll-on-Space/Arrow is
  suppressed by a minimal NATIVE inline onkeydown on the root ul, targeted at
  treeitems only — Blazor's preventDefault directive is all-or-nothing per
  element, so on the li it would trap Tab and cancel Enter/Space on consumer
  buttons. No CSP is configured, so the inline handler runs.

BuildVisibleNodes() mirrors RenderNode's visibility rules and must stay in step.

Tests: 31 new bUnit tests in TreeViewKeyboardNavigationTests; the 47 existing
TreeView tests are unchanged and still green. Docs: new keyboard/a11y section in
docs/components/TreeView.md.
2026-08-01 11:28:20 -04:00
Joseph Doherty 8aa6bf2270 fix(audit): populate ParentExecutionId on alarm-triggered script runs
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.

Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:

  ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
    → SetStaticAttributeCommand.SourceExecutionId
    → AttributeValueChanged.SourceExecutionId   (InstanceActor static-write path)
    → AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext

All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.

Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).

Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.

Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
2026-08-01 11:22:07 -04:00
Joseph Doherty 88638d774a feat(cli,management): close area-move and template-folder CLI parity gaps
Two verified-absent parity gaps between the service layer and the CLI /
ManagementActor command surface, both left as follow-ups by the 2026-05-11
design plans.

(1) area move. AreaService.MoveAreaAsync had existed since the deployment
topology page shipped but was reachable only from the Blazor UI. Adds
MoveAreaCommand(AreaId, NewParentAreaId?) to Commons, a ManagementActor
dispatch arm delegating straight to AreaService.MoveAreaAsync (not-found /
self-parent / descendant-cycle / cross-site / name-collision all surface as
the standard curated ManagementCommandException failure response; the service
writes its own "Move" audit row), and the CLI verb `site area move --id
[--parent-id]`. Omitting --parent-id moves the area to the site root, matching
the command's nullable NewParentAreaId. The command carries the SAME any-of
[Designer, Deployer] gate as CreateArea/UpdateArea/DeleteArea (arch-review C6):
re-parenting is the same structural authoring act, exposed on the same two
surfaces. Placed under the existing `site area` group rather than a new
top-level `area` group, alongside its create/update/delete siblings.

(2) template folder verbs. The five folder management commands have been
handled by ManagementActor since the folder-hierarchy plan, but the promised
CLI surface was never written. Adds `template folder
list|create|rename|move|reorder|delete` mapping 1:1 onto ListTemplateFolders /
CreateTemplateFolder / RenameTemplateFolder / MoveTemplateFolder /
ReorderTemplateFolder / DeleteTemplateFolder. --parent-id is omitted to target
the tree root; --direction takes the lowercase literals up/down, validated at
parse time by AcceptOnlyFromAmong (same case-sensitive contract as the audit
--channel/--kind/--status options).

Follow-on updates: the frozen authorization matrix gains its MoveArea entry
(reflection-driven, so a missing entry would have failed CI); CommandTreeTests
pins both new verb sets plus the omit-parent-id-means-root parse behaviour and
registry round-trips; ManagementActorTests covers the MoveArea role gate and
the delegate-to-service success/root/cycle/not-found paths; the CLI README and
Component-ManagementService.md document the new surface (the latter also gained
the previously-undocumented ReorderTemplateFolder); both plan docs' follow-up
lines are marked done.
2026-08-01 11:15:24 -04:00
Joseph Doherty e0851e3e17 docs(register): close row 12 — InstanceConfigure native-alarm-source CSV upload shipped 2026-08-01 11:12:10 -04:00
Joseph Doherty a26d6ba317 feat(central-ui): native-alarm-source CSV import on InstanceConfigure
The Native Alarm Source Overrides card had no bulk affordance — the CSV path
shipped CLI-only (instance native-alarm-source import --file), so the UI could
only retarget one source at a time inline.

Adds a second InputFile on that card, mirroring the attribute importer's UX
(hidden input behind a button-styled label, 512 KB cap, success/error alert
with the per-line error list, toast). Parsing reuses the SHARED
NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the
new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same
batch rules the server enforces in
ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must
resolve, must not be template-locked, and may appear at most once; any error
rejects the whole file and applies nothing.

Semantics match the CLI: merge, not full replace — sources absent from the
file keep their existing override, a blank field keeps the inherited value,
and an all-blank row clears that source's override (equivalent to the CLI's
all-null override row, without leaving a dead row behind).

Persistence reuses the inline editor's path: SaveNativeOverride's upsert body
is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the
import commits the validated batch in one SaveChangesAsync — no new server
method, no duplicated parsing.

Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field
inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source
and parser-error rejection, plus structural pins on the InputFile wiring.
Docs: Component-CentralUI.md native-alarm-source card gains the import bullet.
2026-08-01 11:11:40 -04:00
Joseph Doherty 697be0ce43 docs(topology): fix stale keep-oldest reference in Site Cluster Behavior — clusters run auto-down since 2026-07-21 2026-08-01 11:10:48 -04:00
Joseph Doherty 4558bc3b1f docs(register): track 4 untracked deferrals, split the failover/perf row, close #18
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).

Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.

Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
2026-08-01 11:10:19 -04:00
Joseph Doherty 2d03f2d507 fix(site-runtime): reconcile artifact deletions on apply — central deletes no longer orphan site rows
The artifact apply (DeploymentManagerActor.HandleDeployArtifacts) was
upsert-only, so deleting an external system (or shared script, DB connection,
data connection) centrally never removed the site's SQLite row — a deleted
external system stayed callable from site scripts forever. Central always
ships the COMPLETE set of each artifact class (ArtifactDeploymentService
GetAll* snapshots; the wire's presence-tracking wrapper lists preserve
null-vs-empty), so the site now applies upsert-then-reconcile: after storing
the incoming set, SiteStorageService.DeleteRowsExceptAsync removes any stored
row absent from it, per artifact table. A null list still means 'field not
shipped' and touches nothing.

Runtime cleanup rides along: a reconciled-away shared script is unregistered
from the compiled SharedScriptLibrary (a stale delegate would stay callable
until restart), and a removed data connection is evicted from the DCL hash
cache and its live connection actor stopped via the previously-caller-less
RemoveConnectionCommand — both on the actor thread via the extended
ApplyArtifactDataConnectionsToDcl message. All four tables are
RegisterReplicated, so the deletes reach the standby as ordinary CDC row
tombstones.

Tests: storage-level reconcile per table (incl. empty-set-deletes-all and
idempotency) in ArtifactStorageTests; actor-level pins in
DeploymentManagerActorTests (orphan delete, null-set no-op, library
unregistration, DCL stop for the removed connection only). Docs:
Component-DeploymentManager + Component-SiteRuntime record the
full-set/reconcile semantics.
2026-08-01 10:54:18 -04:00
Joseph Doherty 0123b68719 docs(plans): MES alarm-status API — all 7 open design questions decided
Design review 2026-08-01: (1) MES relevance = dedicated severity band 900-999
(script constants, real IsFlaggedForMES predicate); (2) MESReceiver version
DROPPED — CvdReactor is the only implementation, router returns a clean
not-supported error elsewhere; (3) Description = Message else AlarmTypeName;
(4) enrich the native mirror NOW with an additive AckTime (proto +
native_alarm_state + DCL stamping); (5) script names match the endpoints;
(6) shared ReactorAlarms on both sides, suffix behavior as planned;
(7) existing MES API key, no new roles. Plan is now ready to execute.
2026-08-01 10:29:44 -04:00
Joseph Doherty d1ade5653b docs(plans): bookkeeping sync — reconcile stale trackers with merged code
A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:

- Flip ~380 stale task statuses to completed across March/May/June/July
  trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
  LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
  code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
  HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
  template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
  (premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
  -> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
  scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
  merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
  sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
  TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
  #28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
  seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
  requirements-traceability 'Pending' clarified as frozen plan-generation
  status.

Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.
2026-08-01 08:53:56 -04:00
Joseph Doherty 663d03e89c Merge branch 'fix/cached-telemetry-drain-hot-loop' 2026-07-27 15:50:42 -04:00
Joseph Doherty c1216044a4 docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings
Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the
bootstrap guard' subsection (dark switch, founder/probe semantics, config table,
accepted trade) plus a forward reference from Dual-Node Recovery. Host
appsettings.Central/Site.json carry a disabled BootstrapGuard block with a
_bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover
note added.
2026-07-24 08:57:44 -04:00
Joseph Doherty 9a12826172 docs(plans): OtOpcUa v3 native-alarm B/C live-gate PASS (#14) — no code needed; item C premise obsolete (Gitea #17) 2026-07-24 04:33:20 -04:00
Joseph Doherty dbec3ee263 docs(plans): OtOpcUa v3 raw-path live-gate PASS (#14) — fix validated live, B/C deferred 2026-07-23 16:54:48 -04:00
Joseph Doherty 266f001a2e docs(plans): OtOpcUa v3 dual-namespace cutover scope + phase-2 plan (#14) 2026-07-23 15:22:22 -04:00