Commit Graph

371 Commits

Author SHA1 Message Date
Joseph Doherty f46cc19736 Merge rows 78+72: DeploymentManager singleton instance sweep — starts replication-seeded enabled instances and converges stale revisions through the real HandleDeploy path; one repair attempt per (instance, revision) 2026-08-16 04:17:45 -04:00
Joseph Doherty e606b24a55 fix(siteruntime): instance reconcile sweep closes config/runtime divergence (register rows 78, 72)
The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.

Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.

Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".

Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).

A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.

Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.

Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
  predecessor still terminating, deployment awaiting its persistence/init join), so it
  can never interleave with a real redeploy -- that window has the config stored and no
  actor, which an unguarded sweep would answer by creating a child whose actor name is
  still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
  staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
  self-heal becoming a restart loop for an actor that dies during init, or for a stored
  config the compile gate rejects (a rejection leaves the running instance untouched:
  the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.

A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.

CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.

Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.

SiteRuntime 626/626, Host 505/505, AuditLog 383/383.

Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md
2026-08-16 04:16:06 -04:00
Joseph Doherty d9a35846d9 perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.

MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):

  393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
  Expiring all of them in ONE tick:  285 s, 2,731,947 oplog rows,
  284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
  i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
  the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
  snapshot of all ten replicated tables, bought by a routine retention purge.

  The resident backlog is produced-minus-drained, so a healthy peer blunts it:
  on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
  the problem, not the reassurance — the old safety margin was a function of
  peer throughput and vanished with the peer down. A bounded tick is safe at
  any peer speed.

  Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).

SHIPPED:
  - SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
    escape hatch, negatives rejected). One SHARED budget across the retention
    purge and the cap trim — two independently-capped phases would let a tick
    emit twice the intended burst. 4x margin under MaxOplogRows, which is also
    shared with every other replicated table's ordinary traffic.
  - SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
    <= PurgeInterval). Without it pacing would make retention converge in 11
    DAYS at the daily cadence; with it the worst case converges in ~11 ticks
    (~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
    the peer's drain and re-accumulate the backlog.
  - A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
    so an expired set landing exactly on the budget does not arm a no-op retry.
  - Visibility: rate-limited (15 min) Warning naming the ceiling and the
    accelerated interval, plus an unthrottled counter
    scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
    backlog converging from a site whose event rate outruns one paced tick.
  - The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
    now anchors to the END of the previous purge. Documented.

ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.

Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.

Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.

Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
2026-08-16 04:05:21 -04:00
Joseph Doherty 3dc38fb674 docs(loadtest+register): close row 29 — 1-hour sustain run settles F1 as not-a-leak (working-set slope -28.89 MB/min, one gen-2 collect, 135M updates at 100.0% nominal); raw metrics JSON archived 2026-08-15 20:27:09 -04:00
Joseph Doherty e3ca77a234 docs(register+decision): rows 69/70/78/79/80 — site-a LocalDb corruption recovered, auto-down deprecation drill-verified benign (explicit provider-class wins; re-drill on every Akka bump), replication-seeded configs start no actors (row-67 side door), OrbStack virtiofs hazard, non-null localDb health fields on unreplicated sites 2026-08-15 19:26:37 -04:00
Joseph Doherty caf9f45165 docs(register): close rows 27, 28, 32, 33, 35, 40, 54-57, 61, 63, 66, 67 with evidence; add rows 71-77; record the row-42 seven-script audit result 2026-08-15 18:03:27 -04:00
Joseph Doherty be29bf471e perf(configdb): index DeploymentRecords on the deployment-list sort key (register row 35)
The deployment list orders `DeployedAt DESC, Id DESC` and joins `Instance` to
make SiteId filterable, but the only DeployedAt index was single-column and
ascending, so the server sorted the filtered set on every page request.

Adds `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)` — the exact
sort key, both columns descending, so a forward scan feeds the paged Skip/Take
directly. Id is in the key as the deterministic tie-break: a bulk site deploy
stamps many rows in the same DeployedAt instant, and without it a row can repeat
or vanish across page boundaries. Three reads share the key
(QueryDeploymentListPageAsync, QueryDeploymentSummariesAsync, and the single-row
GetCurrentDeploymentStatusAsync).

DROPS the old `IX_DeploymentRecords_DeployedAt` rather than keeping both. It is
strictly redundant, not merely overlapping: DeployedAt leads both, so every
predicate it served the composite serves too (direction does not affect
seekability — SQL Server scans either way), and the leaf rows are the same width,
because Id is the clustered PK and was therefore already in the old nonclustered
index's leaf as the row locator. All that changes is Id becoming a declared,
ordered key column. Keeping it would maintain a second index on every deployment
insert and status transition for no read benefit.

Honest caveat: because the clustering key is implicitly ordered within a
nonclustered index, the optimizer may already have been able to serve this
ordering by scanning the old index backward. The explicit composite makes it
deterministic and self-documenting rather than dependent on that inference, and
costs nothing to declare.

Migration 20260815165452_AddDeploymentRecordListSortIndex (scaffolded with a real
build, non-empty, snapshot updated) plus the hand-written production script under
docs/plans/sql/ following the AddAuditLogAndNotificationCoveringIndexes
precedent. SystemArtifactDeploymentRecords has its own separate DeployedAt index
and is deliberately untouched.
2026-08-15 13:22:29 -04:00
Joseph Doherty ef876b38c5 chore(docker+register): refuse worktree deploys (relative bind mounts rehome rig data — register row 68); record row 67 (startup reconcile stores config but starts no InstanceActor) + row-48 postscript 2026-08-15 12:18:01 -04:00
Joseph Doherty 002299c400 docs(register): close rows 46, 48-53 with evidence; add rows 54-66 from the fix wave's discoveries 2026-08-15 12:03:21 -04:00
Joseph Doherty 677263af57 feat(transport): fleet-wide bundle-unlock throttle window (register row 52)
The per-IP unlock throttle's counters were a process-local ConcurrentDictionary
on each central node, so the effective fleet cap was N x
MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed
the client a fresh budget) and a node restart cleared the window outright.

The window now lives in the central MS SQL BundleUnlockAttempt table — the HA
store both central nodes already share, so no new infrastructure, no new port,
no consensus protocol:

- Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository.
- ConfigurationDatabase: entity config (one index,
  IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and
  the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune),
  DbSet, DI registration, migration AddBundleUnlockAttemptTable + the
  hand-written idempotent production script under docs/plans/sql/.
- Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow
  (singleton; resolves the scoped repository from its own scope per call, like
  the central purge singletons). BundleUnlockRateLimiter keeps the policy and
  gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it.

Behaviour preserved from row 44: locked bundles spend no hourly budget (the
lockout check still runs first), throttle denials burn no per-bundle strike and
are NOT recorded so they cannot roll the window forward, unattributable clients
share the stricter <unattributed> bucket, and the budget counts attempts rather
than failures. Config keys and their semantics are unchanged, so the rig's 500
override stays valid.

Three deliberate trades, documented in both Transport docs:
- FAILS OPEN. A store error logs a Warning naming the client and cap, and
  permits the attempt. This is a best-effort rate cap over an already
  authenticated, Administrator-gated surface — the availability of legitimate
  imports outranks the cap, and a DB blip must not make the import wizard
  unusable. Falling back to the process-local window was considered and
  rejected: it would still block a legitimate operator's 11th import.
- Approximate. Count-then-insert is not atomic across nodes, so simultaneous
  attempts can overshoot by the number in flight — bounded, self-correcting
  within the window, and not worth provider-specific raw SQL.
- Pruned opportunistically on the unlock path (CAS-claimed at most every 15
  min per process, bounded to 1000 rows, retaining 2x the window, best-effort
  try/catch) rather than by a new background service — Transport owns no
  central scheduler. Same shape as DeploymentService's terminal-record sweep.

Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over
one store enforcing a single combined window, restart durability, denials
burning no budget, sliding release, bucket isolation, store-failure fail-open,
caller-cancellation not swallowed, no-shared-window fallback, prune and its
negative control); ConfigurationDatabase 395 passed (+5).
2026-08-15 11:52:34 -04:00
Joseph Doherty e59fa5ba4c docs(plans): close register rows 43-44 (transport fixes merged); rows 51-53 (lockout fault mapping, per-node throttle scope, session-cap observability) 2026-08-15 06:06:11 -04:00
Joseph Doherty 23390a28ee docs(plans): close register row 45 (outbox per-delivery adapter scope); rows 49-50 (review-resolution sweep, captive-dependency guard) 2026-08-15 05:43:49 -04:00
Joseph Doherty 9cf24a4496 docs(plans): register row 48 — rig has no running instances, live stream probes trafficless (R2 close-out finding) 2026-08-15 04:32:56 -04:00
Joseph Doherty 88425f662c docs: close residual #3 (R2 merged); register row 47 (CTS-dispose bug, fixed in-flight); correct stale MaxBatchSize story in Component-StoreAndForward (R1 finding) 2026-08-15 04:14:49 -04:00
Joseph Doherty 2c6cfdebe7 docs(plans): close residual #8 (R6 merged); register rows 43-46 (import-session leak, dead rate limiter, outbox DbContext sharing, CLI audit-config query) 2026-08-15 04:06:27 -04:00
Joseph Doherty 9a4d50b3e3 docs(plans): close residual #2 (R1 merged, LocalDb 0.3.0); register rows 41-42 (LocalDb test flake, push.sh publish-bug family check) 2026-08-15 03:50:07 -04:00
Joseph Doherty d7591bf500 docs(plans): close residual #6 (R5 merged); register row 40 (wall-clock-sleep absence-assertion test class) 2026-08-15 03:39:18 -04:00
Joseph Doherty 58a6d47c93 docs(plans): close residuals #4 (R3) and #5 (R4); register rows 35-39 for their findings 2026-08-15 03:35:39 -04:00
Joseph Doherty 267d774bfe docs(plans): close residual #9 (R7 merged); register rows 33-34 for R7's findings (oplog backlog observability, purge-burst pacing) 2026-08-15 03:28:23 -04:00
Joseph Doherty 2b74851f96 docs(plans): residual remediation plan — 7 remaining deferred items, one opus agent each 2026-08-15 03:13:51 -04:00
Joseph Doherty f228ac223f docs(plans): register open rows 29-32 for load-test findings (F1 1-hour run, WP-4 residual scope, F3 burst ceiling, F4 test flake) 2026-08-15 03:07:22 -04:00
Joseph Doherty 40eff637a1 Merge branch 'target-scale-load-test' — deferred register #25 delivered: 375k-subscription target-scale load test with measured evidence (absorbs row 50) 2026-08-15 03:01:51 -04:00
Joseph Doherty 94e8301e44 test(loadharness): retain raw resource samples; record findings F4 and the discarded run
HarnessRunResult now carries every ResourceSample, not just the window summaries.
Motivated by finding F1: the definitive run recorded ZERO gen-2 collections across
45M events, so a positive least-squares heap slope cannot be told apart from
gen-2 garbage that was simply never collected. The summary alone cannot settle
that; the series can. The reported run predates this field -- noted as such in the
results doc rather than implied otherwise.

Also records:
  - the second full-scale run was DISCARDED, not reported: a verification build
    overlapped the start of its measurement window, and a contaminated measurement
    is not evidence.
  - finding F4, a pre-existing test-isolation flake in
    QueueDepthGaugeTests.Gauge_TracksBufferedDepth_AcrossEnqueueDrainAndPark
    (fails in a full-suite run, passes in isolation -- shared static gauge carrying
    state across tests). It cannot originate here: this branch changes zero src/
    files vs its base 986e6e7a. Left unfixed deliberately; filed for separate triage.

Verified: full slnx build clean; SiteRuntime 604/604, Communication 691/691 pass;
TargetScaleHarnessSmokeTests passes (78s).
2026-08-15 03:01:17 -04:00
Joseph Doherty da65605e41 docs(plans): WP-4 target-scale load test RESULTS + close register #25 and row 50
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).

11 clean passes, 1 pass with a caveat, 0 failures:
  tag latency  P50 0.88ms  P99 4.57ms  max 37.41ms  (1.1M samples, end-to-end)
  stream       900,675 delivered, 0 dropped at 100 live subscribers
  health       collect+ingest P99 0.31ms, 10/10 sites tracked
  debug view   P99 2.19ms, 264 completed, 0 timeouts
  deploy       500 instances to a site in 2.6s
  cpu          41% of ONE core = 2.9% of the box
  memory       working-set slope +8.83 MB/min

Three findings, reported rather than tuned away:
  F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
     question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
     positive slope ambiguous. The 1-hour run would settle it. Not tuned.
  F2 (informational, by design) a deferred S&F backlog sits for one full
     DefaultRetryInterval (28.9s measured) before anything drains --
     EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
     misread as slow drainage, so drain is reported as two numbers: retry wait,
     then 3,533 msg/s of actual capacity.
  F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
     100.00% with zero drops while a peer lost 197,028/200,000 events entirely
     within its own bounded channel. Mechanism recorded link by link.

Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.

Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
2026-08-15 02:55:59 -04:00
Joseph Doherty 8abebdae33 docs(plans): target-scale load test harness design memo (WP-4 / register #25)
Records harness architecture, the real-vs-faked table with a justification per fake,
metric definitions, falsifiable pass/fail thresholds derived from the WP-4 acceptance
criteria, and the eight deviations from the WP-4 protocol with reasons — including the
1-hour to 20-minute sustained-window shortening (memory reported as a slope so a
shorter window still answers the leak question) and the four [xc-*] criteria this
harness does not cover.
2026-08-15 02:27:47 -04:00
Joseph Doherty 491df111ea fix(dcl): derive tag-resolution health counts from per-tag authoritative state
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).

DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.

Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:

  total    = _instancesByTag.Count   (the per-tag counted set the residual
                                      called for — distinct tags with at least
                                      one subscribing instance)
  resolved = _subscriptionIds.Count  (tags for which the adapter holds a handle)

Two semantic corrections fall out of the derivation:

- A tag whose subscribe failed at CONNECTION level now counts toward the total.
  It was excluded before, yet the reconnect re-subscribe re-issued it from
  _subscriptionsByInstance and booked it as resolved — resolved above total, and
  a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
  HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
  — still O(1), with no parallel count that can disagree about when a handle is
  released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
  also goes: it existed only to pick which scalar to bump; set sizes get
  DataConnectionLayer-020's double-count cases right for free.

Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).

Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.

Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
2026-08-15 02:05:35 -04:00
Joseph Doherty 7804fe7958 docs: arch-review remediation — component docs sweep, execution log, residuals register
Final consistency sweep per plan §6: verified component docs against shipped
WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime
(recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor
references), TemplateEngine (BundleImporter watermark path), DeploymentManager
(phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm
poll, render coalescing), StoreAndForward (rate-limited drop logging), and
ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the
docs/components/ developer-reference set (SiteRuntime, SiteEventLogging,
InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue
for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for
stream graceful-completion reconnect, the required site audit DB path, honest
CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log
records the phase→commit map, gate results, adversarial-review tally, the
three test-flake root causes, and the nine-item residuals register.
2026-08-15 01:30:59 -04:00
Joseph Doherty 5d075f1374 fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies
Six adversarial-review findings in the central SQL/ingest layer.

F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.

F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).

F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.

F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).

F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.

F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.

Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
2026-08-14 23:46:28 -04:00
Joseph Doherty 312216ff2b docs(plans): script pool split design — WP3.1a 2026-08-14 21:47:48 -04:00
Joseph Doherty 6cfb2dd858 docs(plans): site_events volume policy design — WP3.2a 2026-08-14 21:45:23 -04:00
Joseph Doherty d93f96d08a Merge branch 'worktree-agent-a3b474b485c0288de' into arch-review-remediation 2026-08-14 21:15:31 -04:00
Joseph Doherty 5db2a810c0 perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene 2026-08-14 21:07:12 -04:00
Joseph Doherty 1040dc0fcc docs(plans): DCL batch seam design — WP2.1a 2026-08-14 20:30:35 -04:00
Joseph Doherty 600659d579 perf(sql): sweep/KPI covering indexes + sliced notification terminal purge 2026-08-14 19:55:48 -04:00
Joseph Doherty 0b201e410c docs(plans): arch-review remediation plan — multi-phase, subagent-executable program for all 2026-08-14 review findings 2026-08-14 19:23:56 -04:00
Joseph Doherty d4bb4e17f2 docs(plans): AddTemplateScript OOM re-tested on wonder — no longer reproduces (both forms, throwaway template, <1s) 2026-08-12 18:28:05 -04:00
Joseph Doherty d6b1942932 docs(plans): record uptime-matched wonder working-set re-read — 4.5x/2.0x reduction confirmed flat 2026-08-12 18:25:00 -04:00
Joseph Doherty 4a026d256e docs(plans): record wonder-app-vd03 live-gate PASS — leak closed in production (8.3x metadata collapse) 2026-08-12 18:10:02 -04:00
Joseph Doherty 20f2ae05ca docs(plans): record rig live-gate PASS for the resolver cache (+2 objs/compile vs +289 pre-fix) 2026-08-12 17:49:29 -04:00
Joseph Doherty df17b53f08 docs(plans): mark resolver-cache tasks 1-8 complete 2026-08-12 16:50:43 -04:00
Joseph Doherty 901cec9026 docs(plans): script-compile metadata resolver cache plan 2026-08-12 16:30:53 -04:00
Joseph Doherty d14e0ee4b1 fix(ui): gate detail modals on user intent, not on the row resolving
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).

The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.

Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.

Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.

The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
2026-08-11 06:03:06 -04:00
Joseph Doherty 9e243493fb ui: Central UI density/consistency sweep + Theme 0.4.1
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.

The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.

Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.

NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.

Build 0/0; suite green apart from that one intermittent failure.
2026-08-11 05:50:12 -04:00
Joseph Doherty b6f383a225 docs(notifications): record EWS live-gate PASS (rig -> on-prem Exchange, Delivered first attempt) 2026-08-10 07:29:06 -04:00
Joseph Doherty ba994a59c5 docs: final-review nits — test-stub wording, README tech-stack row, UI-audit follow-up 2026-08-10 07:07:00 -04:00
Joseph Doherty 05d0631cdd chore(plans): mark EWS transport plan tasks 10-11 complete 2026-08-10 07:00:12 -04:00
Joseph Doherty 00d8a923af docs(notifications): EWS transport docs; close Q12 as superseded; design-doc corrections from execution reviews 2026-08-10 06:56:56 -04:00
Joseph Doherty 9f4d7d4bcb docs(notifications): EWS email transport implementation plan (11 tasks) 2026-08-10 06:05:04 -04:00
Joseph Doherty b2ea9c6c74 docs(notifications): design EWS email transport for the outbox; close Q12 as superseded
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
2026-08-10 05:56:41 -04:00
Joseph Doherty e697477c1f feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-07 11:39:28 -04:00