Six adversarial-review findings in the central SQL/ingest layer.
F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.
F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).
F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.
F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).
F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.
F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.
Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
SiteLocalDbSetup.OnReady registered all ten replicated tables
unconditionally, so a deliberately unreplicated site node (site-b and
site-c on the rig) carried the full 30-trigger CDC set forever. Every
write to those tables paid two extra INSERTs plus a json_object
serialization of the whole row, inside the caller's own transaction, and
appended to an oplog nothing ever drains. Arch-review finding #5 (High),
repo half; the library half — trigger cleanup API and O(1) backlog — is
WP3.3.
The ten RegisterReplicated calls are now behind a guard on whether the
node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey.
Either key counts, and the OR is load-bearing rather than defensive:
replication is one bidirectional stream that exactly one side dials, so
only the initiator sets PeerAddress. Verified against the rig — site-a
node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey
alone, site-b/site-c have no Replication section at all. Keying on
PeerAddress alone would have stripped capture from every passive node and
silently made each pair converge in one direction only.
The load-bearing ordering documented in the file is preserved: DDL still
precedes registration, and the legacy migrator still runs unconditionally
after it — an unreplicated node must still absorb its pre-Phase-1 files,
and it has no peer for those rows to be invisible to.
Known residual, documented in-file and in the topology guide: a database
file first created by an older build keeps its stale __localdb_* triggers.
The guard decides whether triggers are installed, not whether existing
ones are removed, and the library has no removal API until WP3.3. Moot on
the docker rig, where a schema-change redeploy recreates the volumes.
The inverse is also now documented: enabling replication on a site that
has run without it does not baseline existing rows, since CDC never
recorded them in __localdb_row_version and the snapshot resync streams
from that ledger.
Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and
absence via sqlite_master across all four config shapes (none, ApiKey
only, PeerAddress + ApiKey, and the notification-table exclusion), plus
DDL-still-runs and migrator-still-runs on the unreplicated branch.
SiteLocalDbWiringTests and the integration site-pair harness now
configure an ApiKey — mirroring the rig's passive node — so their
registration and convergence assertions still describe a replicating
node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests
still pass.
ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.
Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.
Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run
ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.
Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.
This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.
AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.
Two changes:
1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
the whole payload and increments AuditRedactionFailure, reusing the existing
safety-net path. "Not configured at all" stays permissive — conflating those
two states is the actual defect, so both are pinned by tests.
2. Precompile off the hot path. The audit-log roadmap specifies patterns are
"precompiled at startup; rejected if compile takes >100ms"; the implementation
had drifted to compiling lazily on first event, which put a wall-clock budget
on a hot path under production load. RegexOptions.Compiled emits IL during
construction, so a busy node could blow the budget on a perfectly valid
pattern. Warm-up now runs at construction and on every options reload. The
residual window between a reload and its warm-up is safe because that path
now fails closed.
Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).
Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.
Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.
Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.
The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.
The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.
Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).
The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.
Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.
Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.
The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.
The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.
Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.
NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.
Build 0/0; suite green apart from that one intermittent failure.
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.
The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.
Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.
Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.
The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.
Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.
4/4. Merging.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.
Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.
Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.
The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.
Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.
Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.
Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:
- Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
it is not scope-prefixed, why it is read-only, why placeholder rows are
included), the full ScriptAlarm shape, AckTime on the enriched
AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
builds its CDC triggers from the column list at registration time).
- Component-DataConnectionLayer.md already carried the AckTime section in the
first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
- Component-ScriptAnalysis.md: accessors returning domain types return the
SAME type on both surfaces, and the trust-model note that a deny-list needs
no entry for a new globals member -- only that its return type resolves in
a permitted namespace.
- Component-InboundAPI.md records the NEGATIVE decision: there is
deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
per-instance and lives on the site's Instance Actor, so the read goes
through a routed site script and the filtering happens where the data is;
central stays a thin router.
- CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
- The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
what was actually built (incl. the two choices that differ from the plan's
"or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
stay open -- they are deployed config and need a live rig.
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.
Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.
Provenance is honest, never fabricated:
- OPC UA A&C supplies a TRUE ack instant, so we now select it:
AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
index 18, APPENDED so the positional reads at 0-17 keep their meaning.
Servers that omit the field fall back to the event's own Time.
- MxAccess Gateway supplies none, so the ack transition's own timestamp is
used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
re-subscribe snapshot restores one from LastTransitionTimestamp rather
than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.
Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.
Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.
Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
Joseph Doherty