ScadaBridge/archreview/06-edge-integrations.md

# Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries) — Round 2

**Date:** 2026-07-12 (round 2; round-1 report dated pre-2026-07-08, baseline commit `b910f5eb`; everything through HEAD `8c888f13` re-reviewed)

## Scope

Same four components as round 1:

- `src/ZB.MOM.WW.ScadaBridge.InboundAPI` — `POST /api/{methodName}`, API-key auth, Roslyn script handlers, parameter/return validation, routed site calls.
- `src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway` — outbound HTTP client (`Call`/`CachedCall`), error classifiers, S&F handoff.
- `src/ZB.MOM.WW.ScadaBridge.NotificationService` + `src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery` — SMTP/OAuth2, SMS (Twilio) adapter, outbox dispatch/retry policy.
- `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier` — Native-AOT `WWNotifier.exe` client (spec: project README + `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`).

## Method

1. Read the round-1 report and `archreview/plans/PLAN-06-edge-integrations.md` (24 tasks; tracker claims 21 findings fixed, 3 deferred/won't-fix, merged to main `3e964ac2`).
2. Verified **every** round-1 finding against current source with file:line evidence — no plan claim taken on trust. Key verifications: the revision-checked handler cache, `InvalidateMethod`/change-bus wiring, deferred DI-scope disposal, the full `TimeoutSeconds` vertical slice (entity → EF migration → artifact contract → site SQLite → client → mgmt → CLI → UI → doc), `{param}` path templates, bounded response buffering, JSON `AuthConfiguration` with legacy fallback, SMS retry-policy wiring, OAuth2 authority/scope, deterministic SMTP selection.
3. Fresh sweep of `git diff b910f5eb..HEAD` scoped to the four projects, their test projects, the two PLAN-06 EF migrations (`20260710075422_AddExternalSystemTimeoutSeconds`, `20260710080636_AddSmtpOAuth2AuthorityScope`), plus the touched shared surfaces (Host `Program.cs`, `ManagementActor.cs`, `StoreAndForwardService.cs` interaction, `ScadaBridgeTelemetry.cs`). Static review only (suites verified green 2026-07-10).

## Verdict

**Round 1:** mature edge layer with risk concentrated in the stale compiled-handler cache (High), two ESG spec gaps (High/Medium), and unbounded response buffering (Medium). **Round 2:** all of that is genuinely fixed and verified in source — 20 of 25 findings fully fixed, 2 partially, 3 deferred as agreed; the only notable residue is one Medium (the script-artifact invalidation bus publishes to zero subscribers while a Host comment claims the Inbound API subscribes) plus four Lows at the edges of otherwise-correct fixes. No Critical or High new findings. This domain is now in the best shape of any round-1 state I compared against.

---

## Round-1 Finding Disposition

| # | Finding (round 1) | R1 severity | Disposition | Evidence |
|---|---|---|---|---|
| S1 | Compiled inbound handler cache stale across failover (+ known-bad mirror) | High | **Fixed (verified)** | `InboundScriptExecutor.cs:31` (`CachedHandler(Script, Handler)` record), `:371-405` per-request ordinal revision check + known-bad script-text compare + recompile-in-place; known-bad keyed name→script `:52`, overwrite-newer `:71-83`; DB row authoritative per doc-comment `:157-168`. Tests: `InboundScriptExecutorStalenessTests.cs:31,49,67,80` |
| S2 | Method timeout abandons script; DI scope disposed underneath it | Medium | **Fixed (verified)** | `InboundScriptExecutor.cs:328` hoisted `handlerTask`; `:473-494` finally defers scope disposal to a `ContinueWith` that observes the fault; orphan gauge `_abandonedExecutions` `:92-100`; counter `ScadaBridgeTelemetry.cs:104`. (Gauge is internal + OTel counter only — not on the health snapshot, which the plan scoped as "promote later".) |
| S3 | ESG API-key `AuthConfiguration` colon-splitting ambiguous | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:594-599` JSON-first sniff (`{`-prefix), apikey JSON form `:603-625`, basic JSON form `:641-658`, legacy colon fallback preserved `:627-637,661-677`, tolerant parser `TryParseJsonAuth` `:701-722`. Spec updated `Component-ExternalSystemGateway.md:40-41` (colon-in-key preserved verbatim). Tests `ExternalSystemClientTests.cs:832,855,880,906` |
| S4 | DelmiaNotifier failover duplicates undocumented (at-least-once) | Medium | **Fixed (verified — docs)** | `docs/plans/2026-06-26-delmia-recipe-notifier-design.md:108-129` new "Delivery semantics: at-least-once" section + MUST-be-idempotent requirement on `(MachineCode, DownloadPath, WorkOrderNumber)`; `DelmiaNotifier/README.md:57-62` |
| S5 | SMS adapter keeps sending after first transient — duplicate amplification | Medium | **Fixed (verified)** | `SmsNotificationDeliveryAdapter.cs:158-165` breaks recipient loop at first transient; roll-up doc `:293-298`. Test `SmsNotificationDeliveryAdapterTests.cs:303` (`MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted`) |
| S6 | `ErrorClassifier.IsTransient(Exception)` treats any OCE as transient | Low | **Fixed (verified)** | `ErrorClassifier.cs:52-57` cancellation-aware overload; token-less overload carries steer-away doc `:29-33`; call site uses it `ExternalSystemClient.cs:343` |
| S7 | First-of-many SMTP configuration nondeterministic | Low | **Fixed (verified)** | `EmailNotificationDeliveryAdapter.cs:78` `OrderBy(c => c.Id).FirstOrDefault()` + multi-row warning `:83-89`; same in retry-policy resolver `NotificationOutboxActor.cs:450-465` |
| S8 | OAuth2 token cache field visibility | Low | **Fixed (verified)** | `OAuth2TokenService.cs:143-152` immutable `TokenValue` record published through a single `volatile` reference; lock-free read `:61-65`, double-check under per-entry lock `:71-75` |
| S9 | Non-JSON content-type with body yields misleading 400 | Low | **Fixed (verified)** | `EndpointExtensions.cs:186-194` returns 415 `UNSUPPORTED_MEDIA_TYPE`; body with no Content-Type still lenient-parsed. (Chunked-body edge → new finding N5.) |
| P1 | Outbound HTTP responses buffered unbounded | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:329-330` `ResponseHeadersRead`; `ReadBodyBoundedAsync` `:451-483` Content-Length fast-fail + streamed cap+1 abort; `ExternalSystemGatewayOptions.MaxResponseBodyBytes` (10 MiB default) `ExternalSystemGatewayOptions.cs:21`, validated `ExternalSystemGatewayOptionsValidator.cs:29`. Test `ExternalSystemClientTests.cs:1047` |
| P2 | Serial per-recipient Twilio POSTs inside a serial sweep | Medium | **Fixed (verified, beyond plan)** | Plan deferred bounded parallelism, but the outbox dispatcher gained it anyway (arch-review-04 stream): `NotificationOutboxActor.cs` `DeliverGatedAsync` with `SemaphoreSlim(ResolvedMaxParallelDeliveries)` + per-delivery DI scope (`NotificationOutboxOptions.cs:69`); combined with S5's first-transient short-circuit the black-hole worst case is bounded to ~1 timeout per notification |
| P3 | Per-delivery TCP+TLS SMTP handshake | Low | **Won't-fix (accepted)** | Deliberate, documented tradeoff; pooling seam still open in `MailKitSmtpClientWrapper` |
| P4 | Startup compiles every inbound method serially | Low | **Fixed (verified)** | `Host/Program.cs:415` `Parallel.ForEach(methods, method => executor.CompileAndRegister(method))` |
| C1 | Spec drift: per-system ESG timeout specified but not implemented | High | **Fixed (verified — full slice)** | Entity `ExternalSystemDefinition.cs:25`; EF `ExternalSystemConfiguration.cs:32`; migration `20260710075422_AddExternalSystemTimeoutSeconds.cs:13-15` (`AddColumn<int>` on `ExternalSystemDefinitions`); additive artifact `ExternalSystemArtifact.cs:9` (`int TimeoutSeconds = 0` trailing); site SQLite `SiteStorageService.cs:180,735-742` + `SiteExternalSystemRepository.cs:39,71`; client honor `ExternalSystemClient.cs:315-317`; mgmt `ManagementActor.cs:1878,1898`; CLI `ExternalSystemCommands.cs:52,93`; UI `ExternalSystemForm.razor`; doc `Component-ExternalSystemGateway.md:44,107`. Test `ExternalSystemClientTests.cs:975` |
| C2 | Spec drift: path templates never substituted | Medium | **Fixed (verified)** | `ExternalSystemClient.cs:500-501` identifier-only `PathParamRegex`; `:525-543` escaped substitution + consumed-set; excluded from body `:295-298` and query `:556`; doc `Component-ExternalSystemGateway.md:49,81`. Test `:1273`. (Buffered-retry edge → new finding N2.) |
| C3 | Inbound error responses lack documented `code` field / SITE_UNREACHABLE as generic 500 | Low | **Fixed (verified)** | `Error(code, message, status)` helper `EndpointExtensions.cs:66-67` used on every branch (`UNAUTHORIZED :126`, `NOT_APPROVED :166`, `UNSUPPORTED_MEDIA_TYPE :191`, `INVALID_JSON :216`, `VALIDATION_FAILED :235`, passthrough `:279-282`); filter bodies `InboundApiEndpointFilter.cs:59,73` (`STANDBY_NODE`/`BODY_TOO_LARGE`); executor codes `InboundScriptExecutor.cs:387,399,451,463,469` incl. `SITE_UNREACHABLE` via typed `SiteUnreachableException` (`RouteHelper.cs:16-20,386-395`); doc table `Component-InboundAPI.md:111-121`. (Classification mechanism → new finding N3.) |
| C4 | OAuth2 SMTP Microsoft-365-only despite broader spec | Low | **Fixed (verified)** | `OAuth2TokenService.cs:48-56,89-96` optional authority (`{tenant}` template) + scope, cache keyed by credentials\|authority\|scope; entity `SmtpConfiguration.cs:34,40`; migration `20260710080636_AddSmtpOAuth2AuthorityScope.cs:13-22`; adapter passthrough `EmailNotificationDeliveryAdapter.cs:198-201`; CLI `NotificationCommands.cs:178-184`; UI `SmtpConfiguration.razor`; doc `Component-NotificationService.md:49-50` |
| C5 | `InboundApiEndpointFilter` freezes options at construction | Low | **Fixed (verified)** | `InboundApiEndpointFilter.cs:27,34,66` `IOptionsMonitor<InboundApiOptions>` read per request |
| U1 | Script trust boundary static-only (no runtime sandbox) | Underdeveloped | **Deferred (accepted)** | Unchanged by design; `ForbiddenApiChecker` still the static gate (`InboundScriptExecutor.cs:213-223`). Logged in the deferred-work register |
| U2 | No cache-coherence mechanism for compiled handlers | Underdeveloped | **Partially fixed** | The revision-check self-heal (S1) is the real fix and works standalone; `InvalidateMethod` seam exists (`InboundScriptExecutor.cs:136-140`) and the ManagementActor delete path uses it (`ManagementActor.cs:2953`). **But** the claimed bus consumer was never wired — see new finding N1 |
| U3 | Per-SMS retry settings are dead fields | Underdeveloped | **Fixed (verified)** | `NotificationOutboxActor.cs:469-489` `ResolveSmsPolicyAsync` reads `SmsConfiguration.MaxRetries/RetryDelay` (SMTP fallback when absent), per-type selection `:591-595`, non-positive clamp `:494-518`; doc `Component-NotificationService.md:67`. Tests `NotificationOutboxActorRetryEmissionTests.cs` |
| U4 | Twilio accept-only + `AccountSid` un-escaped in URI | Underdeveloped | **Partially fixed (as planned)** | SID format regex at config save `ManagementActor.cs:2156-2161` (`^AC[0-9a-fA-F]{32}$`). Status-callback webhook / per-recipient state: deferred as documented (`Component-NotificationService.md:95`). Pre-validation rows are not retro-checked (minor) |
| U5 | Delmia duplicate semantics undocumented; no CI artifact for shipped binary | Underdeveloped | **Fixed (docs) / Deferred (CI, accepted)** | Docs verified under S4; Windows AOT CI artifact deferred — repo has no Windows runner |
| U6 | Test gaps: handler staleness, oversized outbound body, mid-list SMS transient | Underdeveloped | **Fixed (verified)** | `InboundScriptExecutorStalenessTests.cs` (5 tests), `ExternalSystemClientTests.cs:1047` (oversized body), `SmsNotificationDeliveryAdapterTests.cs:303` (mid-list transient) — exactly the three named gaps |
| U7 | API key rotation procedure unwritten | Underdeveloped | **Fixed (verified)** | `Component-InboundAPI.md:75-77+` "Key rotation" section documenting the dual-key create → migrate → disable procedure |

**Disposition counts:** Fixed (verified) 20 · Partially fixed 2 (U2, U4) · Deferred/won't-fix (accepted) 3 (P3, U1, U5-CI) · Not fixed 0 · Regressed 0.

Also verified: T2's DB-authoritative semantics propagated to the management warning strings (`ManagementActor.cs:2979-2980`) and the rewritten Direct SQL Warning (`Component-InboundAPI.md:158-162`); the two EF migrations are non-empty and additive; PLAN-08's eager options validation landed for all three components (`InboundApiOptionsValidator.cs`, `ExternalSystemGatewayOptionsValidator.cs`, `NotificationOptionsValidator.cs`, registered with `ValidateOnStart`, e.g. `Host/Program.cs:298-300`); the DelmiaNotifier code diff since baseline is XML doc-comments only — the binary's behavior is unchanged.

---

## New Findings (Round 2)

### [Medium] N1 — The script-artifact invalidation bus publishes into the void; the Host comment claims a consumer that does not exist

`Host/Program.cs:96-99` registers `InProcessScriptArtifactChangeBus` with the comment "the Inbound API compiled-handler cache **subscribes as the consumer (wired in plan 06)**" — but a repo-wide search finds **zero** `Subscribe` call sites: the only producer is the Transport bundle importer (`BundleImporter.cs:1811`), and nothing consumes `ScriptArtifactsChanged` for ApiMethods, SharedScripts, *or* Templates. The master tracker's Wave-4 summary makes the same claim ("`InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract"). What actually shipped is the seam (`InboundScriptExecutor.InvalidateMethod`, `InboundScriptExecutor.cs:136-140`) plus one direct call on the delete path (`ManagementActor.cs:2953`).

Correctness is **not** broken for the Inbound API — the per-request revision check (S1) is the designed fallback and fully self-heals bundle-imported ApiMethod scripts. But (a) the Host comment and tracker are factually wrong and will misdirect the next person debugging cache coherence; (b) the PLAN-05→PLAN-06 handoff ("when both land, wire the bus to the invalidate seam", tracker §98) is still open; (c) for the other published `ArtifactKind`s there is no consumer and no documented self-heal claim — if any cache ever keys on SharedScript/Template text at central (e.g. a future compile-verdict cache), this silent no-op bus is a trap. Either wire the one-line subscription in the Host composition root (`bus.Subscribe(n => { if (n.Kind == ApiMethod) executor.InvalidateMethod(n.Name); })`) or fix the comment/tracker to say the fallback is the mechanism.

### [Low] N2 — A now-unresolvable path template on a buffered message is classified transient and retried to exhaustion

`DeliverBufferedAsync` deliberately parks deterministic failures (malformed JSON payload → `return false`, `ExternalSystemClient.cs:186-195`), but the **new** `{param}` substitution throws `ArgumentException` for a placeholder with no matching/non-null parameter (`ExternalSystemClient.cs:535-537`), and `ValidateHttpMethod` throws the same type (`:427-436`). Neither is caught in `DeliverBufferedAsync` (`:174-225` catches only `Permanent…`; everything else propagates), and the S&F retry sweep's catch-all treats any exception as transient (`StoreAndForwardService.cs:822-830`). Scenario: a method's path is edited to add `{id}` while calls are buffered — every already-buffered message now deterministically throws, gets counted as a *transient* failure, and burns the full default 50 retries (`StoreAndForwardOptions.cs:21`) before parking with a misleading "transient" `LastError`. Same shape as the JsonException poison fix in the same method; catch `ArgumentException` there and `return false` (park immediately).

### [Low] N3 — `SITE_UNREACHABLE` classification is substring-sniffing on error message text

`RouteHelper.IsUnreachable` (`RouteHelper.cs:391-395`) decides between `SiteUnreachableException` and plain `InvalidOperationException` by matching `"unreachable"` / `"no contact"` in the failure message produced by `IInstanceRouter`/`IInstanceLocator` — a different component. A harmless rewording of those messages silently downgrades the spec'd `SITE_UNREACHABLE` code to `SCRIPT_ERROR` with no test failing at the boundary. A typed reachability flag on the router result (or a shared const for the message) would make the contract explicit. All five throw sites already funnel through `RoutingFailure` (`:189,237,301,352,372`), so the change is localized.

### [Low] N4 — Bounded response read decodes as UTF-8 unconditionally, dropping charset honoring

Round 1's `ReadAsStringAsync` honored the response's declared charset; the replacement `ReadBodyBoundedAsync` ends with `Encoding.UTF8.GetString(...)` (`ExternalSystemClient.cs:482`) regardless of `Content-Type: ...; charset=iso-8859-1`. For a legacy external system replying in a non-UTF-8 single-byte encoding, response text (and error bodies embedded into script-visible messages) becomes mojibake — a subtle behavior regression smuggled in by an otherwise-correct fix. Reading `content.Headers.ContentType?.CharSet` and falling back to UTF-8 restores parity at negligible cost.

### [Low] N5 — The new 415 guard is bypassed by chunked (no Content-Length) bodies

`EndpointExtensions.cs:187` gates the 415 on `ContentLength > 0`; a chunked non-JSON body has null `ContentLength`, skips the 415, then also skips JSON parsing (the parse condition `:206-207` is `ContentLength > 0 || ContentType contains json`), so `body = null` and the caller gets `VALIDATION_FAILED` "missing required parameter" — the same misleading-error class S9 set out to fix, now confined to the chunked edge. Checking `Request.Headers.ContentLength ?? (Request.Body != null && Request.Headers.TransferEncoding.Count > 0)` — or simply keying the 415 on a present non-JSON `ContentType` alone — closes it.

## What's Genuinely Good

- **The S1 fix is the textbook version of itself.** The DB row is authoritative, the marginal cost is one ordinal string compare of a row already in hand, the known-bad cache became revision-aware in the same move (name→failing-script map, `InboundScriptExecutor.cs:52,71-83`), the test seam is explicitly exempt and documented (`:113-126`), and the semantics change (broken save now 500s consistently on every node instead of node-divergent stale success) was propagated to the management warning strings *and* the component doc — code, warning text, and spec all tell the same story.
- **The `TimeoutSeconds` slice is complete in the way this repo demands.** Ten distinct layers (entity → EF → migration → additive wire contract → site SQLite migration+upsert+read → client resolution order → management → CLI → UI → spec) all present and mutually consistent, with the artifact field additive-trailing per the message-evolution rule.
- **Bounded response reading reuses the house pattern.** Content-Length fast-fail plus streamed cap+1 abort, oversize classified permanent (correct: retrying will not shrink it, and buffering into S&F would defeat the cap), timeout token threaded through the body read so "round-trip" means round-trip (`ExternalSystemClient.cs:348-366`).
- **The JSON auth migration is genuinely backward compatible** — `{`-sniff, tolerant parse that falls through to a warn-and-send-unauthenticated path (never throws on hostile config), legacy colon behavior bit-for-bit preserved, and the never-log-the-value discipline maintained on every new branch.
- **Notification retry policy resolution is now channel-typed, clamped, deterministic, and cheap** (resolved once per sweep, selected per notification), and the outbox dispatcher's bounded parallelism uses a per-delivery DI scope so EF `DbContext` non-thread-safety is respected — the concurrency comment block reads like it was written by someone who has been burned before.
- **Docs kept their promises**: the at-least-once section in the Delmia design doc names the exact idempotency key and puts the obligation on the receiving side where it belongs; the key-rotation procedure is written against the actual `sbk_` token design.

## New-Finding Severity Tally (round 2 only)

| Severity | Count |
|---|---|
| Critical | 0 |
| High | 0 |
| Medium | 1 (N1) |
| Low | 4 (N2, N3, N4, N5) |

Round-1 tally for comparison: 2 High, 6 Medium, 9 Low + 7 underdeveloped areas. The domain's residual risk has collapsed to one wiring/documentation gap and four edge-case polish items.