dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)

Browse Source 
Re-ran all 8 domain reviews at HEAD 8c888f13 against the b910f5eb baseline:
every round-1 finding source-verified (168 fixed, 0 regressions, 0 false
claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low),
concentrated in post-baseline code (anti-entropy resync, KPI rollup
backfill, live alarm stream) and seams the fixes exposed.

Headliners: S&F resync predicate inversion can wipe the delivering node's
buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame
size (02-N2); failover drill kills the one node keep-oldest can't survive
(01-N1); unbounded rollup backfill per failover (04-R1); live production
API key in untracked test.txt (08-NF1).

Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board,
P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
2026-07-12 23:52:10 -04:00
parent 8c888f13a0
commit 5bbd7689fa
26 changed files with 7236 additions and 1321 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archreview/06-edge-integrations.md
+72 -111
View File
@@ -1,143 +1,104 @@
# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries)
# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries) — Round 2
Reviewer domain: the four components where ScadaBridge touches external systems.
**Date:** 2026-07-12 (round 2; round-1 report dated pre-2026-07-08, baseline commit `b910f5eb`; everything through HEAD `8c888f13` re-reviewed)
## Scope
Deep source review (every non-generated `.cs` file read) of:
Same four components as round 1:
- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, audit middleware, routed site calls.
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), `DatabaseGateway` (`Connection`/`CachedWrite`), HTTP + SQL error classifiers, S&F handoff.
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + the delivery adapters in `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP (MailKit) wrapper, OAuth2 token service, SMTP/SMS error classifiers, credential redactor, Email/SMS adapters.
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (arg parser, config loader, failover loop, YES/NO reporter).
- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, routed site calls.
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), error classifiers, S&F handoff.
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP/OAuth2, SMS (Twilio) adapter, outbox dispatch/retry policy.
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (spec: project README + `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`).
Specs compared: `docs/requirements/Component-InboundAPI.md`, `Component-ExternalSystemGateway.md`, `Component-NotificationService.md`, `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`. Test projects examined: `tests/ZB.MOM.WW.ScadaBridge.{InboundAPI,ExternalSystemGateway,NotificationService,NotificationOutbox,DelmiaNotifier}.Tests`.
## Method
## Maturity Verdict
1. Read the round-1 report and `archreview/plans/PLAN-06-edge-integrations.md` (24 tasks; tracker claims 21 findings fixed, 3 deferred/won't-fix, merged to main `3e964ac2`).
2. Verified **every** round-1 finding against current source with file:line evidence — no plan claim taken on trust. Key verifications: the revision-checked handler cache, `InvalidateMethod`/change-bus wiring, deferred DI-scope disposal, the full `TimeoutSeconds` vertical slice (entity → EF migration → artifact contract → site SQLite → client → mgmt → CLI → UI → doc), `{param}` path templates, bounded response buffering, JSON `AuthConfiguration` with legacy fallback, SMS retry-policy wiring, OAuth2 authority/scope, deterministic SMTP selection.
3. Fresh sweep of `git diff b910f5eb..HEAD` scoped to the four projects, their test projects, the two PLAN-06 EF migrations (`20260710075422_AddExternalSystemTimeoutSeconds`, `20260710080636_AddSmtpOAuth2AuthorityScope`), plus the touched shared surfaces (Host `Program.cs`, `ManagementActor.cs`, `StoreAndForwardService.cs` interaction, `ScadaBridgeTelemetry.cs`). Static review only (suites verified green 2026-07-10).
This is a mature, unusually well-hardened edge layer. The code shows evidence of multiple hardening passes: enumeration-safe auth responses, bounded body capture with UTF-8-safe truncation, ordered cancellation-vs-timeout catch discrimination repeated consistently across HTTP, SQL, SMTP, and SMS paths, poison-message parking on both S&F retry paths, credential redaction on every error string, and secrets encrypted at rest (`ExternalSystemConfiguration.cs:27-29,83-85`). Error classification matches the spec (5xx/408/429/connection = transient) on all four channels and is centralized in dedicated classifiers with tests. The remaining risk is concentrated in three places: **cache coherence of compiled inbound script handlers across central failover** (a silent-stale-code bug, the one genuinely High-severity defect found), **two acknowledged spec-vs-code gaps in the External System Gateway** (per-system timeout and path templating), and **unbounded buffering of outbound HTTP response bodies**. DelmiaNotifier is small, faithful to its design doc, and fully tested at the logic level.
## Verdict
**Round 1:** mature edge layer with risk concentrated in the stale compiled-handler cache (High), two ESG spec gaps (High/Medium), and unbounded response buffering (Medium). **Round 2:** all of that is genuinely fixed and verified in source — 20 of 25 findings fully fixed, 2 partially, 3 deferred as agreed; the only notable residue is one Medium (the script-artifact invalidation bus publishes to zero subscribers while a Host comment claims the Inbound API subscribes) plus four Lows at the edges of otherwise-correct fixes. No Critical or High new findings. This domain is now in the best shape of any round-1 state I compared against.
---
## Findings — Dimension 1: Stability
## Round-1 Finding Disposition
### [High] Compiled inbound handler cache goes stale across central failover — old script serves silently with HTTP 200
| # | Finding (round 1) | R1 severity | Disposition | Evidence |
|---|---|---|---|---|
| S1 | Compiled inbound handler cache stale across failover (+ known-bad mirror) | High | **Fixed (verified)** | `InboundScriptExecutor.cs:31` (`CachedHandler(Script, Handler)` record), `:371-405` per-request ordinal revision check + known-bad script-text compare + recompile-in-place; known-bad keyed name→script `:52`, overwrite-newer `:71-83`; DB row authoritative per doc-comment `:157-168`. Tests: `InboundScriptExecutorStalenessTests.cs:31,49,67,80` |
| S2 | Method timeout abandons script; DI scope disposed underneath it | Medium | **Fixed (verified)** | `InboundScriptExecutor.cs:328` hoisted `handlerTask`; `:473-494` finally defers scope disposal to a `ContinueWith` that observes the fault; orphan gauge `_abandonedExecutions` `:92-100`; counter `ScadaBridgeTelemetry.cs:104`. (Gauge is internal + OTel counter only — not on the health snapshot, which the plan scoped as "promote later".) |
| S3 | ESG API-key `AuthConfiguration` colon-splitting ambiguous | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:594-599` JSON-first sniff (`{`-prefix), apikey JSON form `:603-625`, basic JSON form `:641-658`, legacy colon fallback preserved `:627-637,661-677`, tolerant parser `TryParseJsonAuth` `:701-722`. Spec updated `Component-ExternalSystemGateway.md:40-41` (colon-in-key preserved verbatim). Tests `ExternalSystemClientTests.cs:832,855,880,906` |
| S4 | DelmiaNotifier failover duplicates undocumented (at-least-once) | Medium | **Fixed (verified — docs)** | `docs/plans/2026-06-26-delmia-recipe-notifier-design.md:108-129` new "Delivery semantics: at-least-once" section + MUST-be-idempotent requirement on `(MachineCode, DownloadPath, WorkOrderNumber)`; `DelmiaNotifier/README.md:57-62` |
| S5 | SMS adapter keeps sending after first transient — duplicate amplification | Medium | **Fixed (verified)** | `SmsNotificationDeliveryAdapter.cs:158-165` breaks recipient loop at first transient; roll-up doc `:293-298`. Test `SmsNotificationDeliveryAdapterTests.cs:303` (`MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted`) |
| S6 | `ErrorClassifier.IsTransient(Exception)` treats any OCE as transient | Low | **Fixed (verified)** | `ErrorClassifier.cs:52-57` cancellation-aware overload; token-less overload carries steer-away doc `:29-33`; call site uses it `ExternalSystemClient.cs:343` |
| S7 | First-of-many SMTP configuration nondeterministic | Low | **Fixed (verified)** | `EmailNotificationDeliveryAdapter.cs:78` `OrderBy(c => c.Id).FirstOrDefault()` + multi-row warning `:83-89`; same in retry-policy resolver `NotificationOutboxActor.cs:450-465` |
| S8 | OAuth2 token cache field visibility | Low | **Fixed (verified)** | `OAuth2TokenService.cs:143-152` immutable `TokenValue` record published through a single `volatile` reference; lock-free read `:61-65`, double-check under per-entry lock `:71-75` |
| S9 | Non-JSON content-type with body yields misleading 400 | Low | **Fixed (verified)** | `EndpointExtensions.cs:186-194` returns 415 `UNSUPPORTED_MEDIA_TYPE`; body with no Content-Type still lenient-parsed. (Chunked-body edge → new finding N5.) |
| P1 | Outbound HTTP responses buffered unbounded | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:329-330` `ResponseHeadersRead`; `ReadBodyBoundedAsync` `:451-483` Content-Length fast-fail + streamed cap+1 abort; `ExternalSystemGatewayOptions.MaxResponseBodyBytes` (10 MiB default) `ExternalSystemGatewayOptions.cs:21`, validated `ExternalSystemGatewayOptionsValidator.cs:29`. Test `ExternalSystemClientTests.cs:1047` |
| P2 | Serial per-recipient Twilio POSTs inside a serial sweep | Medium | **Fixed (verified, beyond plan)** | Plan deferred bounded parallelism, but the outbox dispatcher gained it anyway (arch-review-04 stream): `NotificationOutboxActor.cs` `DeliverGatedAsync` with `SemaphoreSlim(ResolvedMaxParallelDeliveries)` + per-delivery DI scope (`NotificationOutboxOptions.cs:69`); combined with S5's first-transient short-circuit the black-hole worst case is bounded to ~1 timeout per notification |
| P3 | Per-delivery TCP+TLS SMTP handshake | Low | **Won't-fix (accepted)** | Deliberate, documented tradeoff; pooling seam still open in `MailKitSmtpClientWrapper` |
| P4 | Startup compiles every inbound method serially | Low | **Fixed (verified)** | `Host/Program.cs:415` `Parallel.ForEach(methods, method => executor.CompileAndRegister(method))` |
| C1 | Spec drift: per-system ESG timeout specified but not implemented | High | **Fixed (verified — full slice)** | Entity `ExternalSystemDefinition.cs:25`; EF `ExternalSystemConfiguration.cs:32`; migration `20260710075422_AddExternalSystemTimeoutSeconds.cs:13-15` (`AddColumn<int>` on `ExternalSystemDefinitions`); additive artifact `ExternalSystemArtifact.cs:9` (`int TimeoutSeconds = 0` trailing); site SQLite `SiteStorageService.cs:180,735-742` + `SiteExternalSystemRepository.cs:39,71`; client honor `ExternalSystemClient.cs:315-317`; mgmt `ManagementActor.cs:1878,1898`; CLI `ExternalSystemCommands.cs:52,93`; UI `ExternalSystemForm.razor`; doc `Component-ExternalSystemGateway.md:44,107`. Test `ExternalSystemClientTests.cs:975` |
| C2 | Spec drift: path templates never substituted | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:500-501` identifier-only `PathParamRegex`; `:525-543` escaped substitution + consumed-set; excluded from body `:295-298` and query `:556`; doc `Component-ExternalSystemGateway.md:49,81`. Test `:1273`. (Buffered-retry edge → new finding N2.) |
| C3 | Inbound error responses lack documented `code` field / SITE_UNREACHABLE as generic 500 | Low | **Fixed (verified)** | `Error(code, message, status)` helper `EndpointExtensions.cs:66-67` used on every branch (`UNAUTHORIZED :126`, `NOT_APPROVED :166`, `UNSUPPORTED_MEDIA_TYPE :191`, `INVALID_JSON :216`, `VALIDATION_FAILED :235`, passthrough `:279-282`); filter bodies `InboundApiEndpointFilter.cs:59,73` (`STANDBY_NODE`/`BODY_TOO_LARGE`); executor codes `InboundScriptExecutor.cs:387,399,451,463,469` incl. `SITE_UNREACHABLE` via typed `SiteUnreachableException` (`RouteHelper.cs:16-20,386-395`); doc table `Component-InboundAPI.md:111-121`. (Classification mechanism → new finding N3.) |
| C4 | OAuth2 SMTP Microsoft-365-only despite broader spec | Low | **Fixed (verified)** | `OAuth2TokenService.cs:48-56,89-96` optional authority (`{tenant}` template) + scope, cache keyed by credentials\|authority\|scope; entity `SmtpConfiguration.cs:34,40`; migration `20260710080636_AddSmtpOAuth2AuthorityScope.cs:13-22`; adapter passthrough `EmailNotificationDeliveryAdapter.cs:198-201`; CLI `NotificationCommands.cs:178-184`; UI `SmtpConfiguration.razor`; doc `Component-NotificationService.md:49-50` |
| C5 | `InboundApiEndpointFilter` freezes options at construction | Low | **Fixed (verified)** | `InboundApiEndpointFilter.cs:27,34,66` `IOptionsMonitor<InboundApiOptions>` read per request |
| U1 | Script trust boundary static-only (no runtime sandbox) | Underdeveloped | **Deferred (accepted)** | Unchanged by design; `ForbiddenApiChecker` still the static gate (`InboundScriptExecutor.cs:213-223`). Logged in the deferred-work register |
| U2 | No cache-coherence mechanism for compiled handlers | Underdeveloped | **Partially fixed** | The revision-check self-heal (S1) is the real fix and works standalone; `InvalidateMethod` seam exists (`InboundScriptExecutor.cs:136-140`) and the ManagementActor delete path uses it (`ManagementActor.cs:2953`). **But** the claimed bus consumer was never wired — see new finding N1 |
| U3 | Per-SMS retry settings are dead fields | Underdeveloped | **Fixed (verified)** | `NotificationOutboxActor.cs:469-489` `ResolveSmsPolicyAsync` reads `SmsConfiguration.MaxRetries/RetryDelay` (SMTP fallback when absent), per-type selection `:591-595`, non-positive clamp `:494-518`; doc `Component-NotificationService.md:67`. Tests `NotificationOutboxActorRetryEmissionTests.cs` |
| U4 | Twilio accept-only + `AccountSid` un-escaped in URI | Underdeveloped | **Partially fixed (as planned)** | SID format regex at config save `ManagementActor.cs:2156-2161` (`^AC[0-9a-fA-F]{32}$`). Status-callback webhook / per-recipient state: deferred as documented (`Component-NotificationService.md:95`). Pre-validation rows are not retro-checked (minor) |
| U5 | Delmia duplicate semantics undocumented; no CI artifact for shipped binary | Underdeveloped | **Fixed (docs) / Deferred (CI, accepted)** | Docs verified under S4; Windows AOT CI artifact deferred — repo has no Windows runner |
| U6 | Test gaps: handler staleness, oversized outbound body, mid-list SMS transient | Underdeveloped | **Fixed (verified)** | `InboundScriptExecutorStalenessTests.cs` (5 tests), `ExternalSystemClientTests.cs:1047` (oversized body), `SmsNotificationDeliveryAdapterTests.cs:303` (mid-list transient) — exactly the three named gaps |
| U7 | API key rotation procedure unwritten | Underdeveloped | **Fixed (verified)** | `Component-InboundAPI.md:75-77+` "Key rotation" section documenting the dual-key create → migrate → disable procedure |
`InboundScriptExecutor` is a per-node singleton (`ServiceCollectionExtensions.cs:19`) whose `_scriptHandlers` and `_knownBadMethods` caches (`InboundScriptExecutor.cs:26,39`) are mutated only by:
**Disposition counts:** Fixed (verified) 20 · Partially fixed 2 (U2, U4) · Deferred/won't-fix (accepted) 3 (P3, U1, U5-CI) · Not fixed 0 · Regressed 0.
- the node's own startup compile of all methods (`Host/Program.cs:379-388`), and
- `ManagementActor.HandleUpdateApiMethod`/`HandleDeleteApiMethod` → `CompileAndRegister`/`RemoveHandler` (`ManagementActor.cs:2631,2650`) — which run **only on the node hosting the ManagementActor singleton (the active node)**.
Also verified: T2's DB-authoritative semantics propagated to the management warning strings (`ManagementActor.cs:2979-2980`) and the rewritten Direct SQL Warning (`Component-InboundAPI.md:158-162`); the two EF migrations are non-empty and additive; PLAN-08's eager options validation landed for all three components (`InboundApiOptionsValidator.cs`, `ExternalSystemGatewayOptionsValidator.cs`, `NotificationOptionsValidator.cs`, registered with `ValidateOnStart`, e.g. `Host/Program.cs:298-300`); the DelmiaNotifier code diff since baseline is XML doc-comments only — the binary's behavior is unchanged.
The standby central node compiles every method at *its* startup and then never hears about updates. `ExecuteAsync` looks the handler up by **name only** (`InboundScriptExecutor.cs:311`) — no revision/hash check — even though the endpoint fetches the fresh `ApiMethod` row (with the current `Script` text) from the DB on every request (`EndpointExtensions.cs:142`).
---
Concrete failure scenario:
1. Method `GetOrderStatus` updated via CLI on active node A → A recompiles, serves new logic.
2. Weeks later A fails over to B. B still holds the delegate compiled from the script text as of B's process start.
3. B serves the **old** script indefinitely — HTTP 200, correct-looking JSON, `UpdateApiMethod` long ago reported success. Nothing recompiles until B restarts.
## New Findings (Round 2)
The mirror-image variant: a method that was broken at B's startup landed in B's `_knownBadMethods`; it was later fixed via management update (clears the record on A only). After failover, B answers `"Script compilation failed for this method"` for a method that compiles fine.
### [Medium] N1 — The script-artifact invalidation bus publishes into the void; the Host comment claims a consumer that does not exist
This is the cluster-wide generalization of the already-known "direct SQL edit doesn't clear the cache" issue (documented as a Direct SQL Warning, `Component-InboundAPI.md:129-131`) — but here the write went through the *supported* management path and is still lost. A per-request script-hash comparison against the freshly-fetched row would fix failover staleness, direct-SQL staleness, and known-bad staleness in one move at negligible cost (the row is already in hand).
`Host/Program.cs:96-99` registers `InProcessScriptArtifactChangeBus` with the comment "the Inbound API compiled-handler cache **subscribes as the consumer (wired in plan 06)**" — but a repo-wide search finds **zero** `Subscribe` call sites: the only producer is the Transport bundle importer (`BundleImporter.cs:1811`), and nothing consumes `ScriptArtifactsChanged` for ApiMethods, SharedScripts, *or* Templates. The master tracker's Wave-4 summary makes the same claim ("`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract"). What actually shipped is the seam (`InboundScriptExecutor.InvalidateMethod`, `InboundScriptExecutor.cs:136-140`) plus one direct call on the delete path (`ManagementActor.cs:2953`).
Deleted methods are safe (fresh DB lookup returns null → 403); methods *created* after standby startup are safe (lazy compile, `InboundScriptExecutor.cs:314-330`).
Correctness is **not** broken for the Inbound API — the per-request revision check (S1) is the designed fallback and fully self-heals bundle-imported ApiMethod scripts. But (a) the Host comment and tracker are factually wrong and will misdirect the next person debugging cache coherence; (b) the PLAN-05→PLAN-06 handoff ("when both land, wire the bus to the invalidate seam", tracker §98) is still open; (c) for the other published `ArtifactKind`s there is no consumer and no documented self-heal claim — if any cache ever keys on SharedScript/Template text at central (e.g. a future compile-verdict cache), this silent no-op bus is a trap. Either wire the one-line subscription in the Host composition root (`bus.Subscribe(n => { if (n.Kind == ApiMethod) executor.InvalidateMethod(n.Name); })`) or fix the comment/tracker to say the fallback is the mechanism.
### [Medium] Method timeout abandons the running script; its DI scope is disposed underneath it
### [Low] N2 — A now-unresolvable path template on a buffered message is classified transient and retried to exhaustion
On timeout, `handler(context).WaitAsync(cts.Token)` (`InboundScriptExecutor.cs:333`) returns control but does not stop the underlying script task — cancellation is cooperative via `ctx.CancellationToken` only. The `finally` block then disposes the per-execution DI scope (`InboundScriptExecutor.cs:389-393`) while a non-cooperative script (e.g. mid-flight in a poorly-authored loop, or blocked in a routed call that ignores the token) may still be running and holding `InboundDatabaseHelper` → the scoped `IDatabaseGateway`. Consequences: `ObjectDisposedException`s surfacing as unobserved background faults, and — since the inbound API deliberately has no rate limiting (`Component-InboundAPI.md:140`) — repeated timeouts of a slow method accumulate orphaned executions with no counter or cap. Recommend at minimum a metric for abandoned executions, and deferring scope disposal until the handler task actually completes (`ContinueWith` on the abandoned task).
`DeliverBufferedAsync` deliberately parks deterministic failures (malformed JSON payload → `return false`, `ExternalSystemClient.cs:186-195`), but the **new** `{param}` substitution throws `ArgumentException` for a placeholder with no matching/non-null parameter (`ExternalSystemClient.cs:535-537`), and `ValidateHttpMethod` throws the same type (`:427-436`). Neither is caught in `DeliverBufferedAsync` (`:174-225` catches only `Permanent…`; everything else propagates), and the S&F retry sweep's catch-all treats any exception as transient (`StoreAndForwardService.cs:822-830`). Scenario: a method's path is edited to add `{id}` while calls are buffered — every already-buffered message now deterministically throws, gets counted as a *transient* failure, and burns the full default 50 retries (`StoreAndForwardOptions.cs:21`) before parking with a misleading "transient" `LastError`. Same shape as the JsonException poison fix in the same method; catch `ArgumentException` there and `return false` (park immediately).
### [Medium] ESG API-key `AuthConfiguration` colon-splitting is ambiguous — a key containing `:` silently breaks auth
### [Low] N3 — `SITE_UNREACHABLE` classification is substring-sniffing on error message text
`ExternalSystemClient.ApplyAuth` (`ExternalSystemClient.cs:489-499`) interprets `AuthConfiguration` as `"HeaderName:KeyValue"` when it contains a colon, else as a bare key with default header `X-API-Key`. An API key whose *value* legitimately contains a colon (not rare for opaque tokens) is silently misparsed: the prefix becomes a bogus header name and the request goes out mis-authenticated with only a remote 401 to debug from. The Basic branch (`:504-519`) is fine (split-2 keeps colons in the password), but the apikey branch has no escape hatch. Structured JSON config (`{"header":"X-API-Key","key":"..."}`) — which the entity doc-comment (`ExternalSystemDefinition.cs:13`, "JSON-serialized authentication configuration") already implies — would eliminate the ambiguity.
`RouteHelper.IsUnreachable` (`RouteHelper.cs:391-395`) decides between `SiteUnreachableException` and plain `InvalidOperationException` by matching `"unreachable"` / `"no contact"` in the failure message produced by `IInstanceRouter`/`IInstanceLocator` — a different component. A harmless rewording of those messages silently downgrades the spec'd `SITE_UNREACHABLE` code to `SCRIPT_ERROR` with no test failing at the boundary. A typed reachability flag on the router result (or a shared const for the message) would make the contract explicit. All five throw sites already funnel through `RoutingFailure` (`:189,237,301,352,372`), so the change is localized.
### [Medium] DelmiaNotifier failover can duplicate a processed notification
### [Low] N4 — Bounded response read decodes as UTF-8 unconditionally, dropping charset honoring
`HttpRecipeSender.cs:36-45` maps **any** `HttpRequestException` and any non-caller timeout to `ConnectFailed`, which the loop rolls over to the next URL (`Notifier.cs`, `continue`). Both cases include "the server received and possibly processed the request, but the response never arrived" (connection reset after send; slow response exceeding `TimeoutSeconds`). The next node then receives a second `DelmiaRecipeDownload` POST for the same recipe. This matches the design table (`2026-06-26-delmia-recipe-notifier-design.md:97-99` lists timeout as a failover trigger) — so this is a *design-level* at-least-once semantics that is nowhere documented as such; the inbound `DelmiaRecipeDownload` method must be idempotent on `(MachineCode, DownloadPath, WorkOrderNumber)` or duplicates will surface downstream. Worth one paragraph in the design doc and a check of the deployed method script.
Round 1's `ReadAsStringAsync` honored the response's declared charset; the replacement `ReadBodyBoundedAsync` ends with `Encoding.UTF8.GetString(...)` (`ExternalSystemClient.cs:482`) regardless of `Content-Type: ...; charset=iso-8859-1`. For a legacy external system replying in a non-UTF-8 single-byte encoding, response text (and error bodies embedded into script-visible messages) becomes mojibake — a subtle behavior regression smuggled in by an otherwise-correct fix. Reading `content.Headers.ContentType?.CharSet` and falling back to UTF-8 restores parity at negligible cost.
### [Medium] SMS adapter keeps sending after the first transient failure — inflating duplicate texts on retry
### [Low] N5 — The new 415 guard is bypassed by chunked (no Content-Length) bodies
In `SmsNotificationDeliveryAdapter.DeliverAsync` (`SmsNotificationDeliveryAdapter.cs:148-170`), a transient result does not break the loop; the remaining recipients are still attempted. Since *any* transient rolls the **whole notification** up to `Transient` (`RollUp`, `:297-303`) and retry re-texts everyone, each additional number accepted after the first transient is a guaranteed duplicate SMS on the retry. Short-circuiting on the first transient would reduce duplicates at no delivery cost (the retry re-attempts all numbers anyway). The re-send-to-all characteristic itself is documented (`Component-NotificationService.md:102`), but this amplification is avoidable within v1's no-per-recipient-state constraint.
`EndpointExtensions.cs:187` gates the 415 on `ContentLength > 0`; a chunked non-JSON body has null `ContentLength`, skips the 415, then also skips JSON parsing (the parse condition `:206-207` is `ContentLength > 0 || ContentType contains json`), so `body = null` and the caller gets `VALIDATION_FAILED` "missing required parameter" — the same misleading-error class S9 set out to fix, now confined to the chunked edge. Checking `Request.Headers.ContentLength ?? (Request.Body != null && Request.Headers.TransferEncoding.Count > 0)` — or simply keying the 415 on a present non-JSON `ContentType` alone — closes it.
### [Low] `ErrorClassifier.IsTransient(Exception)` treats any `OperationCanceledException` as transient
## What's Genuinely Good
`ErrorClassifier.cs:32-38` classifies bare `OperationCanceledException`/`TaskCanceledException` as transient. Every current call site is protected by ordered catch filters that peel off caller-cancellation first (`ExternalSystemClient.cs:314-327`), so behaviour is correct today — but the predicate is public and a future call site that skips the ordering would buffer caller-cancelled work into S&F. A `cancellationToken` parameter (as `SmtpErrorClassifier.Classify` and `SmsErrorClassifier.Classify` already take) would make it misuse-resistant.
- **The S1 fix is the textbook version of itself.** The DB row is authoritative, the marginal cost is one ordinal string compare of a row already in hand, the known-bad cache became revision-aware in the same move (name→failing-script map, `InboundScriptExecutor.cs:52,71-83`), the test seam is explicitly exempt and documented (`:113-126`), and the semantics change (broken save now 500s consistently on every node instead of node-divergent stale success) was propagated to the management warning strings *and* the component doc — code, warning text, and spec all tell the same story.
- **The `TimeoutSeconds` slice is complete in the way this repo demands.** Ten distinct layers (entity → EF → migration → additive wire contract → site SQLite migration+upsert+read → client resolution order → management → CLI → UI → spec) all present and mutually consistent, with the artifact field additive-trailing per the message-evolution rule.
- **Bounded response reading reuses the house pattern.** Content-Length fast-fail plus streamed cap+1 abort, oversize classified permanent (correct: retrying will not shrink it, and buffering into S&F would defeat the cap), timeout token threaded through the body read so "round-trip" means round-trip (`ExternalSystemClient.cs:348-366`).
- **The JSON auth migration is genuinely backward compatible** — `{`-sniff, tolerant parse that falls through to a warn-and-send-unauthenticated path (never throws on hostile config), legacy colon behavior bit-for-bit preserved, and the never-log-the-value discipline maintained on every new branch.
- **Notification retry policy resolution is now channel-typed, clamped, deterministic, and cheap** (resolved once per sweep, selected per notification), and the outbox dispatcher's bounded parallelism uses a per-delivery DI scope so EF `DbContext` non-thread-safety is respected — the concurrency comment block reads like it was written by someone who has been burned before.
- **Docs kept their promises**: the at-least-once section in the Delmia design doc names the exact idempotency key and puts the obligation on the receiving side where it belongs; the key-rotation procedure is written against the actual `sbk_` token design.
### [Low] First-of-many SMTP configuration is nondeterministic
## New-Finding Severity Tally (round 2 only)
`EmailNotificationDeliveryAdapter.cs:75-76` and `NotificationOutboxActor.ResolveRetryPolicyAsync` (`NotificationOutboxActor.cs:373-375`) both use `GetAllSmtpConfigurationsAsync()` + `FirstOrDefault()`. With more than one SMTP row the selected server and the retry policy depend on repository ordering. Either enforce single-row semantics or add an explicit "active" flag.
| Severity | Count |
|---|---|
| Critical | 0 |
| High | 0 |
| Medium | 1 (N1) |
| Low | 4 (N2, N3, N4, N5) |
### [Low] OAuth2 token cache: unsynchronized field visibility and unbounded growth
`OAuth2TokenService.CacheEntry.Token/Expiry` are plain fields read outside the per-entry lock (`OAuth2TokenService.cs:48-51`) and written inside it (`:93-94`). The races are benign (worst case: one redundant token fetch), but `volatile`/`Volatile.Read` would make it airtight. The cache is also never evicted — bounded in practice by the number of distinct SMTP credential triples, so acceptable.
### [Low] Non-JSON content-type with a body yields a misleading 400
`EndpointExtensions.cs:181-194`: `ContentLength > 0 || ContentType contains "json"` means a `text/plain` or `application/x-www-form-urlencoded` body is force-parsed as JSON and rejected as "Invalid JSON in request body" rather than a 415. Cosmetic, but the error steers the caller at the wrong problem.
## Findings — Dimension 2: Performance
### [Medium] Outbound HTTP responses are buffered unbounded
`ExternalSystemClient.InvokeHttpAsync` uses `client.SendAsync(request, token)` with the default `ResponseContentRead` completion option (`ExternalSystemClient.cs:312`) and then `ReadAsStringAsync` (`:337`) — the *entire* response body is materialized in memory with **no size cap**. `MaxErrorBodyChars` (`:386`) truncates only the string embedded into error messages, *after* the full read. The inbound side caps request bodies at 1 MiB precisely because "no rate limiting… an explicit, modest cap bounds per-request allocations" (`InboundApiOptions.cs:16-19`); the outbound side extends no such protection against a misbehaving or compromised external system streaming hundreds of MB into a site node (which also runs the instance actors). Use `HttpCompletionOption.ResponseHeadersRead` plus a bounded read (the audit middleware's cap+1 pattern is directly reusable).
### [Medium] Serial per-recipient Twilio POSTs inside a serial dispatch sweep
`SmsNotificationDeliveryAdapter` sends one POST per phone number sequentially (`SmsNotificationDeliveryAdapter.cs:148-170`), and the outbox dispatcher delivers each due notification sequentially within a sweep (`NotificationOutboxActor.cs:320-345`). Worst case for one 50-recipient list against a black-holing endpoint: 50 × 30 s = 25 minutes inside one sweep, during which *all* other due notifications (including email) queue behind it and appear "stuck" on the KPI tiles. Bounded parallelism per notification (e.g. `Parallel.ForEachAsync` with 4–8 DOP — Twilio tolerates it) and/or first-transient short-circuit (see Stability above) would bound the sweep. At current scale (small lists) this is latent, not live.
### [Low] Per-delivery TCP+TLS SMTP handshake
`MailKitSmtpClientWrapper` is deliberately one-connection-per-delivery, extensively documented with the factory seam left open for pooling (`MailKitSmtpClientWrapper.cs:12-38`). Correct tradeoff at current volume; noted only as the first knob if notification volume grows.
### [Low] Startup compiles every inbound method serially
`Host/Program.cs:383-387` runs Roslyn compilation per method in a foreach before `app.RunAsync()`. Fine at tens of methods; would stretch central startup (and failover-recovery window) at hundreds. `Parallel.ForEach` or lazy-only compilation are cheap options later.
### [Good] The hot-path hygiene elsewhere is exemplary
Bounded audit capture at the source with `ArrayPool` scratch buffers and cap+1 over-read detection (`AuditWriteMiddleware.cs:442-502`), fire-and-forget audit write with fault observation (`:358-359,380-398`), `EnableBuffering` skipped for bodyless requests (`:169-191`), name-keyed indexed repository lookups on the ESG hot path (`ExternalSystemClient.cs:540-550`), known-bad-method cache preventing per-request Roslyn recompiles with a 1000-entry flood cap (`InboundScriptExecutor.cs:38-63`), and per-system `HttpClient` factory clients with `MaxConnectionsPerServer` scoped to gateway-owned names only — explicitly avoiding process-global handler replacement (`ExternalSystemGateway/ServiceCollectionExtensions.cs:28-44,92-105`). No socket-exhaustion patterns anywhere: all four components use `IHttpClientFactory` (DelmiaNotifier's single-shot process makes its one `new HttpClient` correct).
## Findings — Dimension 3: Conventions & Spec Fidelity
### [High] Spec drift: per-system ESG timeout is specified but not implemented
`Component-ExternalSystemGateway.md:38` ("**Timeout**: Per-system timeout for all method calls") and `:101` are explicit, but `ExternalSystemDefinition` has no timeout field (`Commons/Entities/ExternalSystems/ExternalSystemDefinition.cs`) and the client acknowledges it: "ExternalSystemDefinition has no per-system Timeout field yet, so the configured DefaultHttpTimeout is the effective round-trip limit" (`ExternalSystemClient.cs:300-305`). Every external system — a fast local MES and a slow remote recipe manager alike — shares one global 30 s (`ExternalSystemGatewayOptions.cs:9`). Repo rules say the design doc is the spec and doc+code travel together; either add the column (entity + EF config + migration + CLI/UI) or amend the spec to the global-timeout reality.
### [Medium] Spec drift: path templates (`/recipes/{id}`) are never substituted
The spec's method definition shows a templated relative path (`Component-ExternalSystemGateway.md:43`), but `BuildUrl` (`ExternalSystemClient.cs:431-463`) concatenates the path verbatim and routes parameters exclusively to the query string (GET/DELETE) or JSON body (POST/PUT/PATCH). A method authored with `{id}` in its path literally calls `.../recipes/{id}`. Either implement `{param}` substitution (with the consumed parameter removed from body/query) or strike the example from the spec before someone authors against it.
### [Low] Spec drift: inbound error responses lack the documented `code` field
`Component-InboundAPI.md:91-97` documents failure bodies as `{"error": ..., "code": "SITE_UNREACHABLE"}`; every actual error body is `{ error }` only (`EndpointExtensions.cs:111-113,153-155,191-193,212-214,253-255`) and a routed site-unreachable failure surfaces as a generic 500 "Internal script error" (`InboundScriptExecutor.cs:382-387` swallows the `InvalidOperationException` from `RouteTarget.Call`, `RouteHelper.cs:172-174`). Machine-readable error codes matter to exactly the audience of this API (external integrators); implement or de-spec.
### [Low] OAuth2 SMTP is Microsoft-365-only despite a broader spec
`OAuth2TokenService` hardcodes `https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token` and the `outlook.office365.com/.default` scope (`OAuth2TokenService.cs:73,80`); the spec says "For Microsoft 365 **and other modern SMTP providers** that require OAuth2" (`Component-NotificationService.md:48`). A configurable authority/scope on `SmtpConfiguration` is a small change; until then the spec overpromises.
### [Low] `InboundApiEndpointFilter` freezes its options at construction
The filter is a singleton taking `IOptions<InboundApiOptions>` (`InboundApiEndpointFilter.cs:32-38`), so a live change to `MaxRequestBodyBytes` is ignored until restart — while the sibling `AuditWriteMiddleware` deliberately hot-reads `IOptionsMonitor<AuditLogOptions>` per request (`AuditWriteMiddleware.cs:97,149`). Inconsistent; use the monitor.
### [Good] Convention adherence is otherwise strong
Options pattern with startup validators throughout (`SmsOptionsValidator`, `NotificationOptions`); additive message-contract evolution respected and even documented at the parameter level ("this optional parameter trails the CancellationToken because it was appended additively", `InboundScriptExecutor.cs:243-246`); secrets consistently encrypted at rest via `EncryptedStringConverter` (ESG `AuthConfiguration` + DB `ConnectionString`: `Configurations/ExternalSystemConfiguration.cs:27-29,83-85`; SMTP credentials + Twilio AuthToken: `Configurations/NotificationConfiguration.cs:69,118`); connection strings never leak into error text (`DatabaseGateway.cs:298`, enforced in messages); credential scrubbing shared, not duplicated (`CredentialRedactor.cs`); API-key auth delegated to the shared `ZB.MOM.WW.Auth.ApiKeys` package (`InboundAPI.csproj:34`) with peppered-HMAC constant-time verification per spec (`Component-InboundAPI.md:66` — the package itself is outside this repo and was not independently verified), generic 401 messages, and byte-identical 403s for not-found vs not-in-scope with the DB lookup ordered *after* the in-memory scope check to close the timing oracle (`EndpointExtensions.cs:117-156`).
## Underdeveloped Areas
1. **Script trust boundary is static-only.** `ForbiddenApiChecker` is candid: semantic + reflection-gateway hardening, "not a true sandbox… genuine containment needs a runtime boundary" (`ForbiddenApiChecker.cs:24-29`). Inbound scripts run in-process on the central node with `Microsoft.CSharp` dynamic binder referenced (`InboundScriptExecutor.cs:172-184`). The deferred restricted-`AssemblyLoadContext`/out-of-process option remains the single biggest security-hardening gap on this boundary.
2. **No cache-coherence mechanism for compiled handlers** (the High finding above) — no revision hash, no cluster invalidation message, no recompile-on-activation hook in `IActiveNodeGate`.
3. **Per-SMS retry settings are dead fields.** `SmsConfiguration.MaxRetries/RetryDelay` are persisted and transported but never read; SMTP config governs retry for *all* notification types (`NotificationOutboxActor.cs:358-401`; documented as deferred in `Component-NotificationService.md:65`).
4. **Twilio delivery is accept-only.** No status-callback webhook, no per-recipient delivery state (both documented as out-of-scope/future — `Component-NotificationService.md:93,102`). Also `AccountSid` is interpolated un-escaped into the request URI (`SmsNotificationDeliveryAdapter.cs:131`) — harmless for a valid SID, but a validation regex at config-save time would close the door.
5. **DelmiaNotifier duplicate-delivery semantics undocumented** (Stability finding); AOT publish is Windows-only and the live smoke is manual (`2026-06-26-delmia-recipe-notifier-design.md:133-136,150-151`) — no CI artifact for the actual shipped binary.
6. **Test gaps** (coverage is otherwise broad and genuinely good across all five test projects — endpoint gating, content-type case-sensitivity, schema-`$ref` runtime resolution, failover loop, both adapters, all four classifiers): nothing exercises the failover/staleness scenario for compiled handlers, nothing exercises an oversized outbound response body, and no test covers a multi-recipient SMS list with a mid-list transient (the duplicate-amplification path).
7. **API key rotation** is enable/disable + scope-set only (`ManagementActor.cs:2685+`); a documented dual-key rotation procedure (create → migrate caller → disable) would round out the ops story — the `sbk_<keyId>_<secret>` design supports it, it's just unwritten.
## Prioritized Recommendations
1. **[High] Make the inbound handler cache self-healing.** Store the compiled delegate together with a hash of the script text; on each request compare against the freshly-fetched `ApiMethod.Script` and recompile on mismatch (also purging `_knownBadMethods`). One change fixes failover staleness, direct-SQL staleness, and stale-bad-method staleness simultaneously; the DB row is already fetched per request so the marginal cost is one string hash.
2. **[High] Close the two ESG spec gaps** — per-system `Timeout` column (entity + EF migration + honored in `InvokeHttpAsync`) and either implement `{param}` path templating or amend `Component-ExternalSystemGateway.md`. Per repo rules, doc and code must travel together.
3. **[Medium] Cap outbound response buffering** in `ExternalSystemClient` (`ResponseHeadersRead` + bounded read, reusing the audit-capture pattern).
4. **[Medium] Bound the SMS blast radius**: short-circuit the recipient loop on first transient, and add bounded parallelism (or per-notification concurrency in the dispatch sweep) so one dead endpoint cannot stall the whole outbox.
5. **[Medium] Replace colon-split `AuthConfiguration` parsing with structured JSON** (the entity already claims to be JSON) with a backward-compatible fallback.
6. **[Medium] Handle abandoned inbound executions**: defer DI-scope disposal to actual handler completion, and add an orphaned-execution counter to the health snapshot.
7. **[Low] Document DelmiaNotifier's at-least-once failover semantics** and verify the deployed `DelmiaRecipeDownload` script is idempotent.
8. **[Low] Small consistency fixes**: `IOptionsMonitor` in `InboundApiEndpointFilter`; token parameter on `ErrorClassifier.IsTransient(Exception)`; configurable OAuth2 authority/scope; single-active-SMTP-config enforcement; implement or de-spec the inbound `code` error field.
Round-1 tally for comparison: 2 High, 6 Medium, 9 Low + 7 underdeveloped areas. The domain's residual risk has collapsed to one wiring/documentation gap and four edge-case polish items.