Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):
- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
client waits (F4), the 7-10 row estimate, the doubled central audit write per
routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
quoted: the method form, both CLI --trace-all-calls help strings, the CLI
README, the ApiMethod entity doc, the manual migration script and the two
Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
the tracking-store test race fixed alongside them (86451382).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Flow Debugger (design 2026-09-05 §5.2): bit NOT NULL DEFAULT 0 column + EF migration
20260906034113 with its docs/plans/sql production twin. CreateApiMethodCommand gains
TraceAllCalls=false; UpdateApiMethodCommand gains bool? TraceAllCalls (null = unchanged so an
older CLI never resets it). Management result echoes the flag; CLI create/update expose
--trace-all-calls; README carries the ~7-rows-per-call warning.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
The per-IP unlock throttle's counters were a process-local ConcurrentDictionary
on each central node, so the effective fleet cap was N x
MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed
the client a fresh budget) and a node restart cleared the window outright.
The window now lives in the central MS SQL BundleUnlockAttempt table — the HA
store both central nodes already share, so no new infrastructure, no new port,
no consensus protocol:
- Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository.
- ConfigurationDatabase: entity config (one index,
IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and
the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune),
DbSet, DI registration, migration AddBundleUnlockAttemptTable + the
hand-written idempotent production script under docs/plans/sql/.
- Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow
(singleton; resolves the scoped repository from its own scope per call, like
the central purge singletons). BundleUnlockRateLimiter keeps the policy and
gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it.
Behaviour preserved from row 44: locked bundles spend no hourly budget (the
lockout check still runs first), throttle denials burn no per-bundle strike and
are NOT recorded so they cannot roll the window forward, unattributable clients
share the stricter <unattributed> bucket, and the budget counts attempts rather
than failures. Config keys and their semantics are unchanged, so the rig's 500
override stays valid.
Three deliberate trades, documented in both Transport docs:
- FAILS OPEN. A store error logs a Warning naming the client and cap, and
permits the attempt. This is a best-effort rate cap over an already
authenticated, Administrator-gated surface — the availability of legitimate
imports outranks the cap, and a DB blip must not make the import wizard
unusable. Falling back to the process-local window was considered and
rejected: it would still block a legitimate operator's 11th import.
- Approximate. Count-then-insert is not atomic across nodes, so simultaneous
attempts can overshoot by the number in flight — bounded, self-correcting
within the window, and not worth provider-specific raw SQL.
- Pruned opportunistically on the unlock path (CAS-claimed at most every 15
min per process, bounded to 1000 rows, retaining 2x the window, best-effort
try/catch) rather than by a new background service — Transport owns no
central scheduler. Same shape as DeploymentService's terminal-record sweep.
Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over
one store enforcing a single combined window, restart durability, denials
burning no budget, sliding release, bucket isolation, store-failure fail-open,
caller-cancellation not swallowed, no-shared-window fallback, prune and its
negative control); ConfigurationDatabase 395 passed (+5).
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.
Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.
Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
New KpiRollupHourly entity is the hourly pre-aggregated series backbone folded
from raw KpiSample rows, letting long-range (30d/90d) trend reads scan ~60x fewer
rows. Same (Source, Metric, Scope, ScopeKey) series key as KpiSample plus
HourStartUtc bucket, folded Value, and MinValue/MaxValue/SampleCount fidelity
columns. UNIQUE IX_KpiRollupHourly_Series (upsert key + range-read cover, with the
default [ScopeKey] IS NOT NULL filter suppressed so Global rows participate) and
IX_KpiRollupHourly_Hour for purge. Non-partitioned, [PRIMARY], no DB-role restriction.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.
Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
Code-review finding UI-Med-2: the design doc + delivery adapter treat FromNumber and
MessagingServiceSid as either-or, but the entity ctor, EF schema, UI and CLI all hard-
required FromNumber — so a Messaging-Service-only Twilio config (a normal production
setup) could not be created. Bring the implementation into line with the spec:
- Commons: SmsConfiguration.FromNumber -> string? (ctor fromNumber optional);
UpdateSmsConfigCommand.FromNumber -> string?.
- ConfigurationDatabase: FromNumber.IsRequired(false) + migration SmsFromNumberOptional
(ALTER COLUMN nullable, idempotent; Down backfills '' — harmless, MsgSid keeps it
deliverable) + regenerated model snapshot.
- Transport: SmsConfigDto.FromNumber -> string? (round-trips a Messaging-Service-only config).
- CentralUI: form validation requires AccountSid + at-least-one-of(FromNumber, MsgSid);
nullable create/edit paths; From-number help text.
- CLI: --from-number no longer Required; BuildUpdateSmsConfigCommand validates the either-or.
- Adapter: From branch null-forgiving (guarded by the existing incomplete-config check).
Tests: ManagementActor MsgSid-only persists null FromNumber; CLI MsgSid-only builds +
neither-throws + contract (--from-number not Required); CentralUI MsgSid-only save.
Findings from the per-module code review of the SMS feature (code-reviews/):
- ManagementService (High): UpdateSmsConfig + UpdateSmtpConfig were Designer-gated
while both /notifications/{sms,smtp} pages enforce RequireAdmin — a Designer
blocked in the UI could still rotate a production credential via CLI. Moved both
to the Administrator arm so the actor gate matches the UI.
- ManagementService (Medium): UpdateSmsConfig treated --auth-token "" as a value,
silently clearing the stored Twilio token. Guard on IsNullOrWhiteSpace so empty ==
omitted (SMTP Credentials keeps its null-only guard — empty is valid for no-auth).
- CentralUI (Medium): NotificationLists recipient badge rendered Name <Email>
unconditionally, showing "Name <>" for SMS lists. Now type-aware (phone for SMS).
- ConfigurationDatabase (Medium): AddSmsNotifications.Down() backfilled NULL emails
to '' — silent data loss for SMS-only recipients. Added a pre-drop guard that
refuses rollback while such rows exist.
- NotificationOutbox (Low): SMS body truncation could split a surrogate pair at the
cap boundary; back off one code unit to stay well-formed.
- Commons (Low): NotificationRecipient public ctor name-guard now matches the
ForEmail factory (IsNullOrWhiteSpace). Documented SmsConfiguration.MaxRetries/
RetryDelay as RESERVED (dispatcher reuses the shared SMTP-derived retry policy).
S1 made NotificationRecipient.EmailAddress nullable + added SmsConfiguration
and four INotificationRepository SMS methods, breaking compilation beyond the
intentionally-deferred central NotificationRepository.
Fix 1 (CS8620/CS8604 nullable EmailAddress projections, email-only paths):
- NotificationOutbox EmailNotificationDeliveryAdapter: filter non-null emails
- DeploymentManager ArtifactDeploymentService: filter non-null emails
- Transport EntitySerializer: filter non-null emails into NotificationRecipientDto
Fix 2 (CS0535): stub the four SMS-config methods on SiteRuntime
SiteNotificationRepository (central-only — NotSupportedException, matching the
existing 'Managed via artifact deployment from Central' write-path pattern).
Doc nits: reword NotificationRecipient private ctor and SmsConfiguration.AuthToken
comments.
The central ConfigurationDatabase.NotificationRepository compile break is left
as-is (S2 implements those four methods).
Spec promised a per-script timeout but only the global ScriptExecutionTimeoutSeconds
existed. Add nullable TemplateScript.ExecutionTimeoutSeconds threaded through EF +
flattening (ResolvedScript) to ScriptExecutionActor/AlarmExecutionActor, which use
perScript ?? global for the execution CTS. Includes the EF migration for the new column.
Joseph Doherty