ScadaBridge/archreview/06-edge-integrations.md

Architecture Review 06 — Edge Integrations (Inbound & Outbound Boundaries) — Round 2

Date: 2026-07-12 (round 2; round-1 report dated pre-2026-07-08, baseline commit b910f5eb; everything through HEAD 8c888f13 re-reviewed)

Scope

Same four components as round 1:

• src/ZB.MOM.WW.ScadaBridge.InboundAPI — POST /api/{methodName}, API-key auth, Roslyn script handlers, parameter/return validation, routed site calls.
• src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway — outbound HTTP client (Call/CachedCall), error classifiers, S&F handoff.
• src/ZB.MOM.WW.ScadaBridge.NotificationService + src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery — SMTP/OAuth2, SMS (Twilio) adapter, outbox dispatch/retry policy.
• src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier — Native-AOT WWNotifier.exe client (spec: project README + docs/plans/2026-06-26-delmia-recipe-notifier-design.md).

Method

1. Read the round-1 report and archreview/plans/PLAN-06-edge-integrations.md (24 tasks; tracker claims 21 findings fixed, 3 deferred/won't-fix, merged to main 3e964ac2).
2. Verified every round-1 finding against current source with file:line evidence — no plan claim taken on trust. Key verifications: the revision-checked handler cache, InvalidateMethod/change-bus wiring, deferred DI-scope disposal, the full TimeoutSeconds vertical slice (entity → EF migration → artifact contract → site SQLite → client → mgmt → CLI → UI → doc), {param} path templates, bounded response buffering, JSON AuthConfiguration with legacy fallback, SMS retry-policy wiring, OAuth2 authority/scope, deterministic SMTP selection.
3. Fresh sweep of git diff b910f5eb..HEAD scoped to the four projects, their test projects, the two PLAN-06 EF migrations (20260710075422_AddExternalSystemTimeoutSeconds, 20260710080636_AddSmtpOAuth2AuthorityScope), plus the touched shared surfaces (Host Program.cs, ManagementActor.cs, StoreAndForwardService.cs interaction, ScadaBridgeTelemetry.cs). Static review only (suites verified green 2026-07-10).

Verdict

Round 1: mature edge layer with risk concentrated in the stale compiled-handler cache (High), two ESG spec gaps (High/Medium), and unbounded response buffering (Medium). Round 2: all of that is genuinely fixed and verified in source — 20 of 25 findings fully fixed, 2 partially, 3 deferred as agreed; the only notable residue is one Medium (the script-artifact invalidation bus publishes to zero subscribers while a Host comment claims the Inbound API subscribes) plus four Lows at the edges of otherwise-correct fixes. No Critical or High new findings. This domain is now in the best shape of any round-1 state I compared against.

---

Round-1 Finding Disposition

#	Finding (round 1)	R1 severity	Disposition	Evidence
S1	Compiled inbound handler cache stale across failover (+ known-bad mirror)	High	Fixed (verified)	InboundScriptExecutor.cs:31 (CachedHandler(Script, Handler) record), :371-405 per-request ordinal revision check + known-bad script-text compare + recompile-in-place; known-bad keyed name→script :52, overwrite-newer :71-83; DB row authoritative per doc-comment :157-168. Tests: InboundScriptExecutorStalenessTests.cs:31,49,67,80
S2	Method timeout abandons script; DI scope disposed underneath it	Medium	Fixed (verified)	InboundScriptExecutor.cs:328 hoisted handlerTask; :473-494 finally defers scope disposal to a ContinueWith that observes the fault; orphan gauge _abandonedExecutions :92-100; counter ScadaBridgeTelemetry.cs:104. (Gauge is internal + OTel counter only — not on the health snapshot, which the plan scoped as "promote later".)
S3	ESG API-key AuthConfiguration colon-splitting ambiguous	Medium	Fixed (verified)	ExternalSystemClient.cs:594-599 JSON-first sniff ({-prefix), apikey JSON form :603-625, basic JSON form :641-658, legacy colon fallback preserved :627-637,661-677, tolerant parser TryParseJsonAuth :701-722. Spec updated Component-ExternalSystemGateway.md:40-41 (colon-in-key preserved verbatim). Tests ExternalSystemClientTests.cs:832,855,880,906
S4	DelmiaNotifier failover duplicates undocumented (at-least-once)	Medium	Fixed (verified — docs)	docs/plans/2026-06-26-delmia-recipe-notifier-design.md:108-129 new "Delivery semantics: at-least-once" section + MUST-be-idempotent requirement on (MachineCode, DownloadPath, WorkOrderNumber); DelmiaNotifier/README.md:57-62
S5	SMS adapter keeps sending after first transient — duplicate amplification	Medium	Fixed (verified)	SmsNotificationDeliveryAdapter.cs:158-165 breaks recipient loop at first transient; roll-up doc :293-298. Test SmsNotificationDeliveryAdapterTests.cs:303 (MidListTransient_ShortCircuits_RemainingRecipientsNotAttempted)
S6	ErrorClassifier.IsTransient(Exception) treats any OCE as transient	Low	Fixed (verified)	ErrorClassifier.cs:52-57 cancellation-aware overload; token-less overload carries steer-away doc :29-33; call site uses it ExternalSystemClient.cs:343
S7	First-of-many SMTP configuration nondeterministic	Low	Fixed (verified)	EmailNotificationDeliveryAdapter.cs:78 OrderBy(c => c.Id).FirstOrDefault() + multi-row warning :83-89; same in retry-policy resolver NotificationOutboxActor.cs:450-465
S8	OAuth2 token cache field visibility	Low	Fixed (verified)	OAuth2TokenService.cs:143-152 immutable TokenValue record published through a single volatile reference; lock-free read :61-65, double-check under per-entry lock :71-75
S9	Non-JSON content-type with body yields misleading 400	Low	Fixed (verified)	EndpointExtensions.cs:186-194 returns 415 UNSUPPORTED_MEDIA_TYPE; body with no Content-Type still lenient-parsed. (Chunked-body edge → new finding N5.)
P1	Outbound HTTP responses buffered unbounded	Medium	Fixed (verified)	ExternalSystemClient.cs:329-330 ResponseHeadersRead; ReadBodyBoundedAsync :451-483 Content-Length fast-fail + streamed cap+1 abort; ExternalSystemGatewayOptions.MaxResponseBodyBytes (10 MiB default) ExternalSystemGatewayOptions.cs:21, validated ExternalSystemGatewayOptionsValidator.cs:29. Test ExternalSystemClientTests.cs:1047
P2	Serial per-recipient Twilio POSTs inside a serial sweep	Medium	Fixed (verified, beyond plan)	Plan deferred bounded parallelism, but the outbox dispatcher gained it anyway (arch-review-04 stream): NotificationOutboxActor.cs DeliverGatedAsync with SemaphoreSlim(ResolvedMaxParallelDeliveries) + per-delivery DI scope (NotificationOutboxOptions.cs:69); combined with S5's first-transient short-circuit the black-hole worst case is bounded to ~1 timeout per notification
P3	Per-delivery TCP+TLS SMTP handshake	Low	Won't-fix (accepted)	Deliberate, documented tradeoff; pooling seam still open in MailKitSmtpClientWrapper
P4	Startup compiles every inbound method serially	Low	Fixed (verified)	Host/Program.cs:415 Parallel.ForEach(methods, method => executor.CompileAndRegister(method))
C1	Spec drift: per-system ESG timeout specified but not implemented	High	Fixed (verified — full slice)	Entity ExternalSystemDefinition.cs:25; EF ExternalSystemConfiguration.cs:32; migration 20260710075422_AddExternalSystemTimeoutSeconds.cs:13-15 (AddColumn<int> on ExternalSystemDefinitions); additive artifact ExternalSystemArtifact.cs:9 (int TimeoutSeconds = 0 trailing); site SQLite SiteStorageService.cs:180,735-742 + SiteExternalSystemRepository.cs:39,71; client honor ExternalSystemClient.cs:315-317; mgmt ManagementActor.cs:1878,1898; CLI ExternalSystemCommands.cs:52,93; UI ExternalSystemForm.razor; doc Component-ExternalSystemGateway.md:44,107. Test ExternalSystemClientTests.cs:975
C2	Spec drift: path templates never substituted	Medium	Fixed (verified)	ExternalSystemClient.cs:500-501 identifier-only PathParamRegex; :525-543 escaped substitution + consumed-set; excluded from body :295-298 and query :556; doc Component-ExternalSystemGateway.md:49,81. Test :1273. (Buffered-retry edge → new finding N2.)
C3	Inbound error responses lack documented code field / SITE_UNREACHABLE as generic 500	Low	Fixed (verified)	Error(code, message, status) helper EndpointExtensions.cs:66-67 used on every branch (UNAUTHORIZED :126, NOT_APPROVED :166, UNSUPPORTED_MEDIA_TYPE :191, INVALID_JSON :216, VALIDATION_FAILED :235, passthrough :279-282); filter bodies InboundApiEndpointFilter.cs:59,73 (STANDBY_NODE/BODY_TOO_LARGE); executor codes InboundScriptExecutor.cs:387,399,451,463,469 incl. SITE_UNREACHABLE via typed SiteUnreachableException (RouteHelper.cs:16-20,386-395); doc table Component-InboundAPI.md:111-121. (Classification mechanism → new finding N3.)
C4	OAuth2 SMTP Microsoft-365-only despite broader spec	Low	Fixed (verified)	OAuth2TokenService.cs:48-56,89-96 optional authority ({tenant} template) + scope, cache keyed by credentials|authority|scope; entity SmtpConfiguration.cs:34,40; migration 20260710080636_AddSmtpOAuth2AuthorityScope.cs:13-22; adapter passthrough EmailNotificationDeliveryAdapter.cs:198-201; CLI NotificationCommands.cs:178-184; UI SmtpConfiguration.razor; doc Component-NotificationService.md:49-50
C5	InboundApiEndpointFilter freezes options at construction	Low	Fixed (verified)	InboundApiEndpointFilter.cs:27,34,66 IOptionsMonitor<InboundApiOptions> read per request
U1	Script trust boundary static-only (no runtime sandbox)	Underdeveloped	Deferred (accepted)	Unchanged by design; ForbiddenApiChecker still the static gate (InboundScriptExecutor.cs:213-223). Logged in the deferred-work register
U2	No cache-coherence mechanism for compiled handlers	Underdeveloped	Partially fixed	The revision-check self-heal (S1) is the real fix and works standalone; InvalidateMethod seam exists (InboundScriptExecutor.cs:136-140) and the ManagementActor delete path uses it (ManagementActor.cs:2953). But the claimed bus consumer was never wired — see new finding N1
U3	Per-SMS retry settings are dead fields	Underdeveloped	Fixed (verified)	NotificationOutboxActor.cs:469-489 ResolveSmsPolicyAsync reads SmsConfiguration.MaxRetries/RetryDelay (SMTP fallback when absent), per-type selection :591-595, non-positive clamp :494-518; doc Component-NotificationService.md:67. Tests NotificationOutboxActorRetryEmissionTests.cs
U4	Twilio accept-only + AccountSid un-escaped in URI	Underdeveloped	Partially fixed (as planned)	SID format regex at config save ManagementActor.cs:2156-2161 (^AC[0-9a-fA-F]{32}$). Status-callback webhook / per-recipient state: deferred as documented (Component-NotificationService.md:95). Pre-validation rows are not retro-checked (minor)
U5	Delmia duplicate semantics undocumented; no CI artifact for shipped binary	Underdeveloped	Fixed (docs) / Deferred (CI, accepted)	Docs verified under S4; Windows AOT CI artifact deferred — repo has no Windows runner
U6	Test gaps: handler staleness, oversized outbound body, mid-list SMS transient	Underdeveloped	Fixed (verified)	InboundScriptExecutorStalenessTests.cs (5 tests), ExternalSystemClientTests.cs:1047 (oversized body), SmsNotificationDeliveryAdapterTests.cs:303 (mid-list transient) — exactly the three named gaps
U7	API key rotation procedure unwritten	Underdeveloped	Fixed (verified)	Component-InboundAPI.md:75-77+ "Key rotation" section documenting the dual-key create → migrate → disable procedure

Disposition counts: Fixed (verified) 20 · Partially fixed 2 (U2, U4) · Deferred/won't-fix (accepted) 3 (P3, U1, U5-CI) · Not fixed 0 · Regressed 0.

Also verified: T2's DB-authoritative semantics propagated to the management warning strings (ManagementActor.cs:2979-2980) and the rewritten Direct SQL Warning (Component-InboundAPI.md:158-162); the two EF migrations are non-empty and additive; PLAN-08's eager options validation landed for all three components (InboundApiOptionsValidator.cs, ExternalSystemGatewayOptionsValidator.cs, NotificationOptionsValidator.cs, registered with ValidateOnStart, e.g. Host/Program.cs:298-300); the DelmiaNotifier code diff since baseline is XML doc-comments only — the binary's behavior is unchanged.

---

New Findings (Round 2)

[Medium] N1 — The script-artifact invalidation bus publishes into the void; the Host comment claims a consumer that does not exist

Host/Program.cs:96-99 registers InProcessScriptArtifactChangeBus with the comment "the Inbound API compiled-handler cache subscribes as the consumer (wired in plan 06)" — but a repo-wide search finds zero Subscribe call sites: the only producer is the Transport bundle importer (BundleImporter.cs:1811), and nothing consumes ScriptArtifactsChanged for ApiMethods, SharedScripts, or Templates. The master tracker's Wave-4 summary makes the same claim ("InvalidateMethod seam consuming PLAN-05's IScriptArtifactChangeBus contract"). What actually shipped is the seam (InboundScriptExecutor.InvalidateMethod, InboundScriptExecutor.cs:136-140) plus one direct call on the delete path (ManagementActor.cs:2953).

Correctness is not broken for the Inbound API — the per-request revision check (S1) is the designed fallback and fully self-heals bundle-imported ApiMethod scripts. But (a) the Host comment and tracker are factually wrong and will misdirect the next person debugging cache coherence; (b) the PLAN-05→PLAN-06 handoff ("when both land, wire the bus to the invalidate seam", tracker §98) is still open; (c) for the other published ArtifactKinds there is no consumer and no documented self-heal claim — if any cache ever keys on SharedScript/Template text at central (e.g. a future compile-verdict cache), this silent no-op bus is a trap. Either wire the one-line subscription in the Host composition root (bus.Subscribe(n => { if (n.Kind == ApiMethod) executor.InvalidateMethod(n.Name); })) or fix the comment/tracker to say the fallback is the mechanism.

[Low] N2 — A now-unresolvable path template on a buffered message is classified transient and retried to exhaustion

DeliverBufferedAsync deliberately parks deterministic failures (malformed JSON payload → return false, ExternalSystemClient.cs:186-195), but the new {param} substitution throws ArgumentException for a placeholder with no matching/non-null parameter (ExternalSystemClient.cs:535-537), and ValidateHttpMethod throws the same type (:427-436). Neither is caught in DeliverBufferedAsync (:174-225 catches only Permanent…; everything else propagates), and the S&F retry sweep's catch-all treats any exception as transient (StoreAndForwardService.cs:822-830). Scenario: a method's path is edited to add {id} while calls are buffered — every already-buffered message now deterministically throws, gets counted as a transient failure, and burns the full default 50 retries (StoreAndForwardOptions.cs:21) before parking with a misleading "transient" LastError. Same shape as the JsonException poison fix in the same method; catch ArgumentException there and return false (park immediately).

[Low] N3 — SITE_UNREACHABLE classification is substring-sniffing on error message text

RouteHelper.IsUnreachable (RouteHelper.cs:391-395) decides between SiteUnreachableException and plain InvalidOperationException by matching "unreachable" / "no contact" in the failure message produced by IInstanceRouter/IInstanceLocator — a different component. A harmless rewording of those messages silently downgrades the spec'd SITE_UNREACHABLE code to SCRIPT_ERROR with no test failing at the boundary. A typed reachability flag on the router result (or a shared const for the message) would make the contract explicit. All five throw sites already funnel through RoutingFailure (:189,237,301,352,372), so the change is localized.

[Low] N4 — Bounded response read decodes as UTF-8 unconditionally, dropping charset honoring

Round 1's ReadAsStringAsync honored the response's declared charset; the replacement ReadBodyBoundedAsync ends with Encoding.UTF8.GetString(...) (ExternalSystemClient.cs:482) regardless of Content-Type: ...; charset=iso-8859-1. For a legacy external system replying in a non-UTF-8 single-byte encoding, response text (and error bodies embedded into script-visible messages) becomes mojibake — a subtle behavior regression smuggled in by an otherwise-correct fix. Reading content.Headers.ContentType?.CharSet and falling back to UTF-8 restores parity at negligible cost.

[Low] N5 — The new 415 guard is bypassed by chunked (no Content-Length) bodies

EndpointExtensions.cs:187 gates the 415 on ContentLength > 0; a chunked non-JSON body has null ContentLength, skips the 415, then also skips JSON parsing (the parse condition :206-207 is ContentLength > 0 || ContentType contains json), so body = null and the caller gets VALIDATION_FAILED "missing required parameter" — the same misleading-error class S9 set out to fix, now confined to the chunked edge. Checking Request.Headers.ContentLength ?? (Request.Body != null && Request.Headers.TransferEncoding.Count > 0) — or simply keying the 415 on a present non-JSON ContentType alone — closes it.

What's Genuinely Good

• The S1 fix is the textbook version of itself. The DB row is authoritative, the marginal cost is one ordinal string compare of a row already in hand, the known-bad cache became revision-aware in the same move (name→failing-script map, InboundScriptExecutor.cs:52,71-83), the test seam is explicitly exempt and documented (:113-126), and the semantics change (broken save now 500s consistently on every node instead of node-divergent stale success) was propagated to the management warning strings and the component doc — code, warning text, and spec all tell the same story.
• The TimeoutSeconds slice is complete in the way this repo demands. Ten distinct layers (entity → EF → migration → additive wire contract → site SQLite migration+upsert+read → client resolution order → management → CLI → UI → spec) all present and mutually consistent, with the artifact field additive-trailing per the message-evolution rule.
• Bounded response reading reuses the house pattern. Content-Length fast-fail plus streamed cap+1 abort, oversize classified permanent (correct: retrying will not shrink it, and buffering into S&F would defeat the cap), timeout token threaded through the body read so "round-trip" means round-trip (ExternalSystemClient.cs:348-366).
• The JSON auth migration is genuinely backward compatible — {-sniff, tolerant parse that falls through to a warn-and-send-unauthenticated path (never throws on hostile config), legacy colon behavior bit-for-bit preserved, and the never-log-the-value discipline maintained on every new branch.
• Notification retry policy resolution is now channel-typed, clamped, deterministic, and cheap (resolved once per sweep, selected per notification), and the outbox dispatcher's bounded parallelism uses a per-delivery DI scope so EF DbContext non-thread-safety is respected — the concurrency comment block reads like it was written by someone who has been burned before.
• Docs kept their promises: the at-least-once section in the Delmia design doc names the exact idempotency key and puts the obligation on the receiving side where it belongs; the key-rotation procedure is written against the actual sbk_ token design.

New-Finding Severity Tally (round 2 only)

Severity	Count
Critical	0
High	0
Medium	1 (N1)
Low	4 (N2, N3, N4, N5)

Round-1 tally for comparison: 2 High, 6 Medium, 9 Low + 7 underdeveloped areas. The domain's residual risk has collapsed to one wiring/documentation gap and four edge-case polish items.