Commit Graph

796 Commits

Author SHA1 Message Date
Joseph Doherty f07ba5cff5 docs(deployment): QA vq03/vq04 + prod vd03 upgraded to afb27e77 (template picker) — binary-only, no migrations; all six health endpoints Healthy
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 08:34:49 -04:00
Joseph Doherty d6a31f1cd0 docs(plans): template picker — live rig gate PASS (New Instance…, ?templateId prefill, picker reveal/search/cancel, Disable verb) + task status
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 08:07:22 -04:00
Joseph Doherty 4bb77cf0f7 docs(centralui): template picker on Create Instance, New Instance… from the Templates tree
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 08:01:37 -04:00
Joseph Doherty 5b4a27b23e feat(centralui): Create Instance picks its template from the folder tree (+ ?templateId preselect)
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 07:42:14 -04:00
Joseph Doherty 22468c1548 docs(plans): template picker design + 8-task plan — Create Instance picks from the Templates folder tree (TemplatePickerDialog, ?templateId preselect), Templates tree gains Deployer-gated 'New Instance…', Topology Disable confirm verb pinned (already fixed by 0db253b5)
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 06:51:27 -04:00
Joseph Doherty a3e2df325c docs(deployment): QA vq03/vq04 + prod vd03 upgraded to 691a2f2f (create-time bindings) — binary-only, no migrations; healthy; outgoing-process stop-timeout FTL noted
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 06:36:24 -04:00
Joseph Doherty 691a2f2f80 docs(plans): create-time bindings — live rig gate PASS (Object-node pick, prefill, 30/30 bound, deploy-gate A/B) + task status
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 06:20:50 -04:00
Joseph Doherty a26edffd4c docs: create-time connection bindings (Central UI, DeploymentManager preview flatten, CLI cross-ref)
Claude-Session: https://claude.ai/code/session_01G2GHPnbbKGvyyfbr2XBRVH
2026-08-21 06:14:55 -04:00
Joseph Doherty a7301bb638 docs(plans): create-time connection bindings — design (inline card on Create Instance, object-node Browse, land on Configure) + 9-task plan
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-21 05:59:00 -04:00
Joseph Doherty 8beee522a3 docs(deployment): prod vd03 hotfix da0bfe2b (binary-only swap via WinRM, app.bak.20260820T170309) — healthy, boot-time advise retries match the a24fa38e boot
Claude-Session: https://claude.ai/code/session_01Rs5tk49eMrNcwmi8FAohHX
2026-08-20 17:08:27 -04:00
Joseph Doherty 50f7439e33 docs(deployment): QA hotfix da0bfe2b — MxGateway graceful-stream-end disconnect; incident root cause (dashboard session close 15:29:03) 2026-08-20 16:57:45 -04:00
Joseph Doherty da0bfe2be1 fix(dcl): MxGateway adapter treats a graceful end of the event stream as a disconnect — a dashboard 'close session' (or graceful gateway shutdown) completes the stream with OK, leaving the adapter Connected on a dead session: values frozen at last GOOD quality, every write 'Session … was not found' until restart (QA vq03 Z28061 MoveIn, 2026-08-20). Pinned: signals once while connected, silent for superseded loops / own teardown 2026-08-20 16:51:13 -04:00
Joseph Doherty f5a74db3de docs(deployment): record prod upgrade of wonder-app-vd03 to a24fa38e (via WinRM from vq04) — 6 EF migrations, AuditLog:SiteWriter path, System32 auditlog.db relocated; Z28061 270 rows / 8 groups 2026-08-20 16:40:00 -04:00
Joseph Doherty 5b9cf772db docs(deployment): record QA upgrade of wonder-app-vq03 + vq04 (all 4 nodes) to a24fa38e — 6 EF migrations, AuditLog:SiteWriter path added, System32 auditlog.db relocated; grouped bindings verified on Z28061 (264 rows / 8 groups) 2026-08-20 16:31:57 -04:00
Joseph Doherty a24fa38eae docs(plans): grouped connection bindings — all 9 tasks complete 2026-08-20 16:12:13 -04:00
Joseph Doherty 546250096d fix(centralui): grouped bindings review fixes — Apply with the header at none/mixed keeps each member's connection (never unbinds; upsert Save could not persist it), object inference runs over effective references (bare-default members make an overridden group mixed, default-only prefixes infer), bindings to a deleted connection count as unbound, 'n changed · unsaved' diffs against last-saved state; bUnit behaviour tests for fallback banner, grouping and mixed-group Apply+Save 2026-08-20 16:09:34 -04:00
Joseph Doherty 5e04e81ba0 docs: grouped connection bindings — Central UI Configure page, ResolvedAttribute.DefaultDataSourceReference, CLI instance bind-module; correct the stale 'set-bindings REPLACES all' claim (server upserts) in README + help text 2026-08-20 16:01:33 -04:00
Joseph Doherty 4cfd9337a3 docs(plans): grouped connection bindings implementation plan (9 tasks) 2026-08-20 15:49:36 -04:00
Joseph Doherty eef77b6654 docs(plans): grouped connection bindings design — Configure page sources binding rows from the flattened config (composed members were invisible), per-module Object bulk apply, CLI instance bind-module parity 2026-08-20 15:44:40 -04:00
Joseph Doherty 4388fa41e4 deps(localdb)+docs: consume LocalDb 0.3.4 — snapshot as_of_seq floors at last_acked_seq, ending the full-snapshot-on-every-reconnect loop (register row 106 closed, live-proven on the rig: second pair restart exchanged zero snapshots) 2026-08-20 10:00:37 -04:00
Joseph Doherty 25c15dd164 ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.

Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.

Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
2026-08-20 09:37:48 -04:00
Joseph Doherty 09989fdf32 deps(localdb)+docs: consume LocalDb 0.3.3 (FK-parent registration guard + atomic per-entry apply, arch-review P4-7); Low-wave integration docs — register rows 103 closed / 104 auto_vacuum legacy files / 105 FK-parent constraint; CLAUDE.md bulk-deploy up-to-date filter + --force, 0.3.3 line, infra/LDAP correction (P8-7); plan doc status 2026-08-20 08:56:50 -04:00
Joseph Doherty 1cea01d924 fix(notifications): list sort index + CAS operator Retry/Discard (P5-3, P5-4)
P5-3: add IX_Notifications_Created_NotificationId (CreatedAt DESC,
NotificationId DESC) — the Central UI Notification Outbox list page sorts on
exactly that pair, but no existing index leads unconditionally on CreatedAt,
so an unfiltered/lightly filtered page fell back to a full clustered-index
scan + sort on every COUNT + page request. Mirrors the Medium wave's
IX_SiteCalls_Created_TrackedOperationId migration + SQL twin pattern
faithfully, including the handwritten docs/plans/sql script.

P5-4: operator Retry/Discard was read-check-write keyed on NotificationId
only — a detached PipeTo task interleaving with the dispatcher's own write
(or a second concurrent operator action) could land between the read and
the write and get silently clobbered by the unconditional UpdateAsync,
stamping a Delivered row Discarded with stale fields. Adds
INotificationOutboxRepository.TryUpdateFromParkedAsync, a single-statement
compare-and-swap (WHERE Status == Parked) mirroring
SecuredWriteRepository.TryMarkApprovedAsync's CAS shape. NotificationOutboxActor's
RetryAsync/DiscardAsync now use the CAS write and, on a lost race, re-read
to report "notification already progressed to <Status>" instead of silently
overwriting whatever is there now.

Proof of failure: reverting only the actor's CAS usage while keeping the
repository addition reproduces the silent-overwrite bug — 7 tests fail,
including two new deterministic (non-sleep) race-proof tests that mock the
exact GetByIdAsync -> lost-CAS -> re-read sequence the finding describes.
P5-3 is perf-only; verified via migration/SQL-twin parity instead (no
runtime proof-of-failure is meaningful for an index).
2026-08-19 16:41:04 -04:00
Joseph Doherty 942acc37b4 docs(plans): low-findings remediation plan — 21 review Lows + register row 103 as 5 phases / 15 model-matched tasks 2026-08-19 16:01:03 -04:00
Joseph Doherty 9b67c7cb4b docs(plans)+docs(register): 2026-08-19 Medium-findings remediation plan (all 17 tasks executed); reopen+re-close register rows 95/99 with the P8-6/P6-3 defect proofs; add row 103 (deferred P6-6 lesser instances); CLAUDE.md — bulk deploy skips Disabled, LocalDb 0.3.2 2026-08-19 15:07:24 -04:00
Joseph Doherty 013257ec7c fix(centralui): per-IP spray ceiling on LoginThrottle (arch-review 2026-08-19 P6-9)
LoginThrottle keyed failures only as {username}|{ip}, so a password spray
from one IP against many distinct usernames never accumulated
MaxLoginFailuresPerWindow failures on any single pair and was never
throttled, despite the class doc claiming spray protection.

Add a secondary per-IP counter (SecurityOptions.PerIpMaxFailures, default
25, validated strictly greater than MaxLoginFailuresPerWindow) that shares
the existing fixed window/lockout machinery but tracks failures by IP
alone. IsLockedOut now refuses a bind when either the per-pair or the
per-IP counter is locked. RecordSuccess clears only the per-pair counter
(never the per-IP one), so a spray that stumbles onto one valid account
cannot use that success to reset its IP budget and keep spraying; window
expiry is the only relief for the per-IP counter. Per-pair behavior for
normal (non-spray) flows is unchanged.

Updates the class doc and Component-Security.md to describe the two-tier
model instead of the single-tier spray claim.
2026-08-19 14:54:57 -04:00
Joseph Doherty 5cbc143b01 fix(deploymentmanager): bulk deploy skips Disabled + reconcile circuit-breaker + probe captures pre-flatten versions (arch-review 2026-08-19 P7-2 + P7-3 + P7-4) 2026-08-19 14:52:33 -04:00
Joseph Doherty 64e6ba70e8 fix(management): site-scope ListDataConnections + hoist scope checks above resolve/status oracles (arch-review 2026-08-19 P6-4 + P6-5)
P6-4: HandleListDataConnections took no user at all and returned EVERY
site's connections to any authenticated caller, nullifying the row-98
fail-closed gate on HandleGetDataConnection — Get denied exactly what List
handed over. DataConnectionPublicShape already scrubs secrets, so the leak
was site topology plus a connection-id existence oracle, but it still
defeated scoping. List now takes the AuthenticatedUser: an explicit SiteId
goes through EnforceSiteScope (the ListAreas shape), and an unfiltered list
from a site-scoped non-Administrator caller is narrowed to their permitted
sites (the ListSites shape). GetRequiredRoles is untouched — scoping is not
a role gate, so the frozen 138-command authz matrix is unaffected.

The Central UI Design > Connections page bypasses the actor entirely
(ISiteRepository.GetAllDataConnectionsAsync direct) and IS reachable by a
scoped user: it is gated RequireDesign, and SiteScopeService explicitly does
NOT exempt Designer (register row 86). It now filters sites through
SiteScope.FilterSitesAsync and keeps only their connections, mirroring
EventLogs/AlarmSummary.

P6-5: four handlers resolved their target — and on approve/reject reported
its status — BEFORE the scope check, so an out-of-scope caller could tell a
real id from a fake one, and read a secured write's lifecycle status, from
the error shape alone. The scope check is hoisted above the not-found and
status checks in HandleSubmitSecuredWrite, HandleApproveSecuredWrite,
HandleRejectSecuredWrite and HandleDebugSnapshot, with the unresolvable case
routed through the existing DenyUnresolvedTargetIfScoped so the denial is
identical whether or not the target exists. System-wide callers keep their
ordinary not-found/status answers, and the P6-1 CAS semantics on
approve/reject are unchanged (only the checks above the CAS moved).

Tests: 8 new ManagementService tests + 2 CentralUI bUnit tests, all verified
to FAIL against the unfixed handlers (src reverted, tests re-run) and pass
after — including the unscoped/Administrator control cases proving behaviour
for system-wide callers is unchanged. Full ManagementService.Tests 545 green,
CentralUI.Tests 1073 green.
2026-08-19 14:49:24 -04:00
Joseph Doherty eca0ab768d fix(auditlog)+fix(configdb): budget the ingest fallback loop; index + varchar-type the SiteCalls list query (arch-review 2026-08-19 P5-1 + P5-2)
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.

P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.

Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
2026-08-19 14:49:24 -04:00
Joseph Doherty 6c11362d9f fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.

P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.

Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
2026-08-19 14:49:24 -04:00
Joseph Doherty d18f344e66 docs(plans): remediation plan for the four 2026-08-19 arch-review High findings (P2-1, P2-2, P6-1, P6-2) — all four executed in this session 2026-08-19 14:01:23 -04:00
Joseph Doherty 61a14bccab fix(management): CAS-guard secured-write reject (WHERE Status='Pending') so reject can never falsify an executed two-person write (arch-review 2026-08-19 P6-1 High)
Reject read the row as Pending, then landed a blind full-row UpdateAsync. A
concurrent approve could win its own Pending->Approved CAS in that gap and enter
the multi-second WriteTagAsync relay; the reject's update then stamped
Status/ExecutedAtUtc/ExecutionError to Rejected mid-relay, so the two-person-control
record said Rejected for a write that physically executed at the plant. Works
cross-node (both centrals share one MS SQL).

- ISecuredWriteRepository/SecuredWriteRepository: new TryMarkRejectedAsync,
  mirroring TryMarkApprovedAsync exactly (conditional UPDATE ... WHERE
  Status='Pending', writes only the columns the reject transition owns). No schema
  change, no migration, no rowversion column.
- HandleRejectSecuredWrite: uses the CAS; on loss re-reads the row and returns the
  status that actually won (same UX shape as approve's lost-race error), and emits
  the SecuredWriteReject audit row ONLY on a won CAS.
- Audited the other lifecycle writes: approve/reject/expiry (inline + the
  opportunistic list sweep) now ALL transition only out of Pending, so winning the
  approve CAS gives that leg exclusive ownership of the row and its post-relay
  Executed/Failed update stays safe as an unconditional write. Documented in place.
- Tests: repository-level approve-vs-reject races in both orders (exactly one wins,
  loser false, row reflects the winner only) plus reject-vs-expiry; handler-level
  lost-race reject (conflict error, no write, no audit row) and approve on an
  already-rejected row (no CAS, no relay, no audit).
2026-08-19 13:54:45 -04:00
Joseph Doherty e9e44a9ef0 test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale
The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
2026-08-18 10:32:46 -04:00
Joseph Doherty d3be5f0527 test(communication): close register row 102 — serialize the two gauge-moving classes so the subscribers.active baseline holds
The gauge test polls baseline+1, but the process-global gauge's baseline is
captured once: a parallel class's streams closing after capture lowers the
gauge below baseline+1 forever. Survey confirmed only SiteStreamGrpcServerTests
and CleanupVerificationTests drive the real SubscribeInstance handler that
moves the counter; both now share the SiteStreamSubscriberGauge collection
(row-96 pattern). 4/4 full-suite runs green, durations unchanged.
2026-08-18 09:18:43 -04:00
Joseph Doherty 324ba369d9 infra(docker)+test(siteeventlog): close register tripwires 79 + 93 — rig data to named volumes; pin site_events schema to the cap-model calibration
Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.

Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
2026-08-18 08:43:52 -04:00
Joseph Doherty 5936573e71 docs(register): close row 38 — OtOpcUa racing-clear test rebuilt on retry-exhaustion assertion cross-session (34829ea0, 837d8a17) 2026-08-18 08:08:17 -04:00
Joseph Doherty 4cf71e92a5 docs(register): close row 39 — OtOpcUa conditional LocalDb registration + baseline/deregister self-heal ported cross-session (3e62f161, df533916) 2026-08-18 07:44:08 -04:00
Joseph Doherty b3af16dbac chore(deps): bump ZB.MOM.WW.LocalDb to 0.3.1 — sync-session fault-teardown handshake-flush fix; rig redeployed, site-a replication Connected/backlog 0 2026-08-18 07:43:08 -04:00
Joseph Doherty 7eb47e878b docs(register): close rows 41+42 — fixed cross-session in scadaproj (3e5a5fd LocalDb 0.3.1 handshake-flush fix, b62981f seven pack/push scripts anchored) 2026-08-18 07:21:28 -04:00
Joseph Doherty 226a628f6f docs(register): row-101 postscript — whole-snapshot generation guard shipped (all three instance-config hazards proven real: backward restart via old-ConfigJson redeploy, delete/disable resurrection, boot name-set skew; both-ends bump corrects the completion-only ordering argument); add row 102 (stream-gauge fixed-baseline flake) 2026-08-18 05:59:51 -04:00
Joseph Doherty f02e116688 fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:

- Row-72 leg: a read dispatched before a deploy's persist, landing after the
  deploy completed AND after the pipeline released ownership, carries the
  revision the deploy replaced. TryRestartStaleInstance then "repairs" the
  instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
  InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
  per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
  enabled row while the actor is already stopped, so StartStoredInstance
  resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.

Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).

_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.

CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.

A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.

Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.

Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
2026-08-18 05:38:00 -04:00
Joseph Doherty 4da5f7cd7a docs(register): close row 101 — stale-snapshot race verified real (worse than guessed: spurious removal is NOT self-healing), definitions-write generation guard shipped in 0d3a90a9 2026-08-17 07:04:40 -04:00
Joseph Doherty 0d3a90a9fd fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101
An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
2026-08-17 07:03:26 -04:00
Joseph Doherty 63d7f273e6 docs(register): Wave 6 close-out — close rows 97 (in-place DCL reconfigure, live-probed round trip), 98 (fail-closed scope gate, 5 sites not 2 — id-enumeration oracle closed), 99 (strict audit filters, 7 gates + lax-server premise corrected, live-probed), 100 (compile cache instance-scoped, Clear() deleted — zero production callers); row-88 postscript (ordinal guard); add row 101 (overlay-sweep race hypothesis) 2026-08-17 06:44:41 -04:00
Joseph Doherty 8a9a8d2951 merge(agent-c): row 100 — SiteScriptCompileCache becomes an instance-scoped sealed class owned by the ScriptCompilationService DI singleton (zero production Clear() callers existed, so Clear() is deleted outright); row-91 serialized collection relaxed and delta assertions restored to exact counts; new registration test pins the load-bearing singleton lifetime 2026-08-17 06:20:17 -04:00
Joseph Doherty 152ec85e92 merge(agent-b): rows 98/99 — site-scope gate fail-closed on unresolvable targets (5 sites, not 2: the inline GetInstance/GetSite/GetDataConnection shapes were an id-enumeration oracle; all route through DenyUnresolvedTargetIfScoped, denial indistinguishable from out-of-scope); CLI audit query/export drop 7 case-sensitive AcceptOnlyFromAmong gates for help-text values + a new strict server-side ParseEnumListStrict (no silent drop, no ordinals) 2026-08-17 06:20:14 -04:00
Joseph Doherty 7c1268d85c feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.

New `ReconfigureConnectionCommand` (Commons, same shape as the create):

- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
  per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
  reconfigure — and logs "Reconfigured DCL connection {Connection}
  (protocol={Protocol}) in place" at Information in place of the old Warning. The
  winner-resolution / arbitration / tombstone logic is untouched; only what
  happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
  actor, falling back to a create for a name it does not hold, so the command is
  safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
  connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
  which is exactly when a correction arrives; stashed it would wait on the broken
  configuration forever. Handling replaces protocol/primary/backup/failover-count,
  re-arms endpoint selection to Primary (the new config's primary is the intent,
  whatever the old config had failed over to), and calls the new shared
  `SwapAdapter` helper — now used by both failover sites too, so the swap exists
  once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
  _adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
  reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
  racing attempts) with the unstable-disconnect accounting and failover check
  suppressed — the link was dropped on purpose, and counting it as an unstable
  cycle could trip a failover onto the NEW backup before the new primary had been
  tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
  the untouched subscription registries. Idempotent: a reconfigure carrying the
  configuration already in force is a no-op.

Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.

Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.

Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
2026-08-17 06:19:13 -04:00
Joseph Doherty f33f25a10b fix(auth+cli): fail-closed site-scope gate (row 98); case-insensitive audit filter values with a strict server-side check (row 99)
Register row 98 — the site-scope authorization gate was FAIL-OPEN on an
unresolvable target. The resolve-then-check shape was written as
`if (target != null) EnforceSiteScope(...)`, so a null resolve skipped the
check entirely and a site-scoped caller passed the gate for any id in the
system. Authorization now fails CLOSED: a target that cannot be PROVEN to sit
inside the caller's scope is treated as outside it.

The defect turned out to have THREE more instances than the register recorded.
Besides `EnforceSiteScopeForInstance` and `EnforceSiteScopeForIdentifier`, the
same inline shape sat in `HandleGetInstance`, `HandleGetSite` and
`HandleGetDataConnection` — read handlers that resolve their own target and so
never call the helpers. Those were the sharper half of the bug: they answered
success-with-null for a nonexistent id and Unauthorized for an out-of-scope
one, making the pair an id-enumeration oracle over the whole fleet. All five
now route through one shared `DenyUnresolvedTargetIfScoped` arm.

Two properties are load-bearing and pinned by tests:
  - System-wide callers (Administrator, or a Deployer with no site grants) are
    untouched — they return at the guards above the lookup and keep the
    handler's ordinary "not found" / null answer.
  - The denial is indistinguishable from an out-of-scope denial: same
    ManagementUnauthorized shape, and a message that never states whether the
    target exists.

Register row 99 — CLI `audit query --status delivered` failed client-side while
the server would have accepted it: `AcceptOnlyFromAmong` matches ordinally
against the capitalised literals. Same treatment row 88 gave `deploy status` —
the client-side gate is removed, the valid set moves into the option help text,
and the server becomes the single authoritative check.

That required ADDING the server check, not just relying on one: the shared
audit filter parser was LAX, silently dropping an unrecognised value and
leaving the dimension unconstrained — so removing the CLI gate alone would have
turned a loud client-side failure into a plausible, complete-looking page that
did not answer the question asked. New `ParseEnumListStrict` (case-insensitive,
rejects ordinals — `Enum.TryParse` accepts "3", and a filter is a name-based
contract) backs the ManagementService /api/audit/query + /api/audit/export
endpoints, which now return 400 BAD_REQUEST naming the valid set.

Other AcceptOnlyFromAmong sites surveyed in the audit group: `--channel`,
`--kind`, `--status` on BOTH `query` and `export` (6 uses) plus `export
--format` (rejected `--format CSV` though the server lowercases it) — all seven
fixed here. Deliberately left alone: `template folder reorder --direction`
(pure client-side literal, no server-side parse to defer to) and the global
`--format json|table` (CLI-local output selection, never sent to a server).

The CentralUI AuditLogPage drill-in parser and the CentralUI export endpoint it
feeds deliberately KEEP the lax parse — they read a query string a human may
have bookmarked, where a stale fragment must still render the page. Both
contracts now live side by side in AuditQueryParamParsers and are documented as
non-reconcilable.

Docs: Component-ManagementService.md gains a "Site-scope gate: fail-closed on
an unresolvable target" section and a strict-filter note on the audit
endpoints; Component-Security.md records the fail-closed shape under Deployer
site scoping; the CLI README marks the audit filter values case-insensitive and
states the no-silent-drop / no-ordinals contract.

Tests: 5 new ManagementActorTests fail before the row-98 fix (verified by
reverting ManagementActor.cs) and cover scoped+nonexistent -> denied,
system-wide+nonexistent -> unchanged (both Administrator and no-grant Deployer),
scoped+in-scope -> allowed, and the indistinguishability property. 21 CLI tests
fail before the row-99 fix. Four pre-existing tests that pinned the removed
client-side gate were rewritten to pin the new contract.
2026-08-17 06:05:29 -04:00
Joseph Doherty 84b405ff5f refactor(siteruntime): scope SiteScriptCompileCache to the ScriptCompilationService singleton — close register row 100
The compile cache was a `static` class, so its entry table and its Hits/Count
counters were one process-global blob every test class in the SiteRuntime
assembly wrote into. Row 91 last wave neutered the resulting flakes (per-test
GUID keys + one-sided counter bounds), but the shared mutable state itself
remained: any test calling Clear() evicted entries other parallel classes had
just inserted, and a Clear() from a class outside the serialized collection
would degrade the others to recompiles and re-open the row-91 flake class.

PREMISE CORRECTION: the register speculated production clears "per-instance or
per-deployment script entries" and offered ClearKeys(prefix) as the natural fit.
There are NO production callers of Clear() at all — grep of src/ finds only
GetOrAdd (ScriptCompilationService.CompileCore). Clear() existed solely so
tests could reset the global. A key-scoped API would therefore have been new
production surface built for nobody, so the instance-scoped seam is the fix and
Clear() is deleted outright rather than migrated.

- SiteScriptCompileCache: `internal static class` -> `internal sealed class`;
  Cache/EvictionLock/_hits/_accessSequence become instance fields. LRU batch
  eviction, the double-checked sweep and the ConcurrentDictionary.ToArray()
  snapshot are byte-for-byte the same policy. Clear() removed.
- ScriptCompilationService owns one cache. Public ctor (the DI path) builds its
  own; a new internal ctor takes one for tests. Production N4 sharing is
  unchanged and now rests on the DI SINGLETON registration — the one service
  instance is threaded to DeployCompileValidator and to every Instance Actor —
  so that lifetime is load-bearing and is pinned by a new registration test.
- Tests: all three Clear() callers migrated off it. Each test now owns its cache,
  which lets the assertions get STRONGER — the row-91 workarounds are retired:
  Hits/Count are exact again (no >= baseline bounds), the per-test GUID key
  token is gone, and ForeignEntrySlack (512 entries of allowance on the bound)
  drops to the racing-thread count in the concurrency test. New test
  SeparateCaches_DoNotShareEntriesOrCounters pins the scoping directly: filling
  one cache past MaxEntries leaves another's entry, hit count and identity
  intact. DeploymentWarmThenGateTests additionally asserts Count == 1 now that
  its counter is private.
- The [Collection("SiteScriptCompileCache")] serialization from row 91 is
  RELAXED (removed from all three classes) — they no longer touch any shared
  state, so there is nothing left to serialize.
- Docs: Component-SiteRuntime.md gains the ownership/singleton note and drops
  the Clear() reference; DeploymentManagerActor / DeployCompileValidator /
  ServiceCollectionExtensions comments corrected to say where the cache lives.

Build: 0 warnings. SiteRuntime.Tests 643/643, three consecutive full-parallel
runs. AuditLog.Tests correlation classes (other ScriptCompilationService
consumers) green.
2026-08-17 06:03:58 -04:00
Joseph Doherty dcfcf822af docs(register): Wave 5 close-out — close rows 37/86/87/88 (deployment-read consolidation + Administrator system-wide fix, live-probed), 89/90 (DCL definitions-overlay arbitration, premise corrected twice, live-probed), 91/95/96 (flake family + parallelism policy); add rows 97-100 (live-connection reconfigure gap, fail-open scope gate, case-sensitive audit --status, process-global compile-cache Clear) 2026-08-17 05:41:56 -04:00