ScadaBridge/stillpending.md

# ScadaBridge — Pending / Deferred / Partial / Missing Functionality Audit

**Date:** 2026-06-15
**Scope:** Full system — design specs (`docs/requirements/`), all of `src/`, the Central UI / CLI / Management Service surfaces, and the plan/checklist archive (`docs/plans/`).
**Method:** Five parallel read-only investigators, each verifying doc claims against actual code (file:line evidence). Top findings were independently corroborated by 2+ agents.

## Executive summary

The codebase is unusually clean: **zero real `TODO`/`FIXME` markers in `src/`**, and all 11 implementation phases self-report complete. Consequently the unfinished work does **not** announce itself — it hides in three forms:

1. **Silent gaps (Tier 1)** — documented as working, not marked deferred, but absent or inert in production.
2. **Partial / behaviorally-divergent functionality (Tier 2)** — real, but narrower or different from the spec.
3. **Intentional deferrals (Tier 3)** — knowingly punted, correctly documented, with extensible seams. *Not defects.*
4. **Doc↔code drift (Tier 4)** — code is fine; the specs describe a superseded architecture.

**Actionable risk is concentrated in Tier 1.** Recommended starting point: **#3 / #4 (wire the two never-started audit actors)** — highest impact, smallest blast radius.

---

## Tier 1 — Silent gaps: documented as working, but not actually running

These are the dangerous ones. Specs present them as live behavior, they are **not** marked deferred, yet the functionality is absent or inert.

| # | Gap | Where | Impact |
|---|-----|-------|--------|
| **1** | **Script "test-compilation" does no real compilation.** The headline pre-deploy gate ("scripts must compile without errors") is a brace-balance + forbidden-API *substring* scan. No Roslyn reference in the project. | `TemplateEngine/Validation/ScriptCompiler.cs:56-104`; used by `ValidationService.cs:128` | Semantically-broken C# passes validation and **deploys**. Found independently by 2 agents. |
| **2** | **Script forbidden-API gate is bypassable.** The script trust model's only design-time enforcement is the same substring scan — defeated by aliases / `using static` / `global::`. Self-documented as "SECURITY LIMITATION (TemplateEngine-006)". | `ScriptCompiler.cs:14-22,61-72`; `ValidationService.cs:346` | Security boundary is advisory only. 2 agents. |
| **3** | **Audit Log 365-day retention purge never starts.** `AuditLogPurgeActor` exists but has **zero** `ActorOf`/`Props.Create` callers; only the roll-*forward* partition service runs. | `AuditLog/.../AuditLogPurgeActor.cs:58`; `AkkaHostedService.cs:486` ("wired in a later bundle") | The documented purge does not run in production; `AuditLog` grows unbounded. |
| **4** | **Audit Log reconciliation self-heal never wired.** `IPullAuditEventsClient` has no implementation; `SiteAuditReconciliationActor` is never instantiated. | `IPullAuditEventsClient.cs:31`; `SiteAuditReconciliationActor.cs:68` | The documented "lost-telemetry fallback" doesn't exist; forward telemetry is the only path. |
| **5** | **Site Call Audit reconciliation pull + daily purge both missing.** The actor's own docstring admits it. `PurgeTerminalAsync` is implemented but never invoked. | `SiteCallAuditActor.cs:28-34`; `SiteCallAuditRepository.cs:213` | `SiteCalls` mirror has no self-heal and grows unbounded. |
| **6** | **Site Event Logging emits only 2 of 7 documented categories.** Only `connection` + `script` (error-path only). Alarm, Deployment, Store-and-Forward, Instance-Lifecycle, Notification events are never logged — `ISiteEventLogger` isn't injected into those subsystems. | `DataConnectionActor.cs`, `ScriptExecutionActor.cs` (only emitters); spec "Events Logged" §20-28 | Operational event log is materially incomplete vs spec. |

---

## Tier 2 — Partial / behaviorally-divergent functionality

Real, but narrower than the spec — wrong in a way that could surprise an operator or script author.

| # | Gap | Where |
|---|-----|-------|
| **7** | **`Database.CachedWrite` misclassifies permanent SQL errors as transient** → retries forever instead of failing fast to the script. The API path does it right; the DB path does not. No immediate attempt, no synchronous permanent-`Failed` return. | `DatabaseGateway.cs:78-204` (cf. `ExternalSystemClient.cs:100-161`) |
| **8** | **Alarm `conditionFilter` is plumbed end-to-end but applied nowhere** — set a filter on a native-alarm source and it silently mirrors *all* conditions. | `DataConnectionActor.cs:1482,1540-1554`; `RealOpcUaClient.cs:242,295`; `MxGatewayDataConnection.cs:154-167` |
| **9** | **Per-script execution timeout doesn't exist** — spec promises per-script; only a global `ScriptExecutionTimeoutSeconds`. No field in the template/flattened model to carry it. | `SiteRuntimeOptions.cs:31`; `ScriptExecutionActor.cs:100`; `AlarmExecutionActor.cs:66` |
| **10** | **Connection-level diffs never surface in the deployment diff** — `ComputeConnectionsDiff` is dead code (no callers); `ConfigurationDiff` has no slot for it. Per-attribute binding drift *is* caught; standalone connection endpoint (protocol/config/failover) diff is not. | `DiffService.cs:158-204`; `Commons/Types/Flattening/ConfigurationDiff.cs:7-24` |
| **11** | **Inbound API auth transport drift** — code uses `Authorization: Bearer sbk_<keyId>_<secret>`; doc says `X-API-Key` header. | `InboundAPI/.../EndpointExtensions.cs:83-90` |
| **12** | **Inbound API audit write is fire-and-forget *after* response flush** — doc says synchronous *before* flush. Row is still emitted (fail-soft), just non-blocking and after the body is forwarded. | `AuditWriteMiddleware.cs:195-212,281-290` |
| **13** | **Inbound `Object`/`List` extended types are shape-validated only** — no nested/field-level type validation, despite spec implying typed/nested validation. | `ParameterValidator.cs:109-145`; `ReturnValueValidator.cs:18` |
| **14** | **JWT-in-cookie session design not implemented** — `/auth/login` signs a plain `ClaimsPrincipal`; `GenerateToken` only used by the CLI `/auth/token` path; `ValidateToken` has no external callers. | `AuthEndpoints.cs:38,75-112,152`; `ServiceCollectionExtensions.cs:99-118` |
| **15** | **"Re-query LDAP every 15 min / roles never >15 min stale" not implemented for interactive sessions** — `JwtTokenService.RefreshToken`/`RecordActivity`/`ShouldRefresh`/`IsIdleTimedOut` have **zero** call sites; roles fixed until cookie expiry. The 15-min sliding + 30-min idle layers are collapsed into a single 30-min sliding cookie window. | `JwtTokenService.*` (no callers); `ServiceCollectionExtensions.cs:99-148` |
| **16** | **Transport stale-instance enumeration always returns empty** — `BundleImporter` returns `Array.Empty<int>()`; UI shows a generic warning with no count, link not filtered to stale instances. | `BundleImporter.cs:733`; `TransportImport.razor:347-388` |
| **17** | **`MachineDataDb` fail-fast requirement not enforced** — spec (REQ-HOST-3/4) requires central nodes to validate a non-empty `MachineDataDb` connection string. `DatabaseOptions` has only `ConfigurationDb`/`SiteDbPath`; validator never checks it; 0 `grep` hits in `src/`. Key lives only in docker appsettings as dead config. | `DatabaseOptions.cs:6-12`; `StartupValidator.cs:60-61` |
| **18** | **CI grep-guard against `UPDATE/DELETE … AuditLog` not in the repo** — spec claims a build-time grep that fails on data-layer mutations. DB-role DENY enforcement *is* present in migrations (so this is a backstop, not the only control), but the claimed code-level guard is absent. | spec `Component-AuditLog.md:335-336`, `Component-ConfigurationDatabase.md:297` |

### Lower-severity Tier-2 / behavioral notes

| # | Gap | Where |
|---|-----|-------|
| 19 | **Script "started"/"completed" events not logged** (only failures, severity `Error`). | `ScriptExecutionActor.cs:239,256`; `ScriptActor.cs:369` |
| 20 | **Return-type compatibility check is dead scaffolding** — `BuildReturnMap` builds maps never read; no return-type comparison runs. | `SemanticValidator.cs:62-63,279-287` |
| 21 | **Argument *type* compatibility not checked** — only arg *count* (comma counting). | `SemanticValidator.cs:251-266,390-425` |
| 22 | **Native-alarm-source connection-capability validation never runs in deploy pipeline** — `alarmCapableConnectionNames` param no production caller supplies. | `SemanticValidator.cs:30-33,239-245`; `FlatteningPipeline.cs:93,115` |
| 23 | **Connection-binding completeness is a non-blocking Warning, not deploy-gating Error**; "name exists at site" half missing. | `ValidationService.cs:504-519`; `ValidationResult.cs:9` |
| 24 | **Debug snapshot/subscribe for unknown instance returns empty snapshot, not error** — caller can't distinguish "not deployed" from "deployed but empty." | `DeploymentManagerActor.cs:845-866` |
| 25 | **Recursion-limit error logged to .NET `ILogger`, not the site event log** as spec requires. | `ScriptRuntimeContext.cs:302-305,464-466` |
| 26 | **Debug-stream snapshot/stream ordering reversed; no timestamp-dedup replay** — `PreStart` sends snapshot first, opens stream after; gap-window events lost (spec wants stream-first + replay/dedup). | `DebugStreamBridgeActor.cs:89-103,163-166` |
| 27 | **OPC UA native-alarm transition leaves several display fields empty** (Category/Description/OperatorUser/OriginalRaiseTime/CurrentValue/LimitValue) — partly by design. | `RealOpcUaClient.cs:395-403`; `MxGatewayAlarmMapper.cs:79-113` |
| 28 | **Readiness gate omits "required cluster singletons running" criterion** — covers membership + DB connectivity only (softened by spec's "(if applicable)"). | `Program.cs:188-201,314-317`; `AkkaClusterHealthCheck.cs:54` |
| 29 | **SiteEventLog active-node purge gate never registered** — `SiteEventLogActiveNodeCheck` not added to DI; purge defaults to `() => true`, runs on standby too (harmless, but documented restriction unenforced). | `SiteEventLogging/ServiceCollectionExtensions.cs:33-37`; `EventLogPurgeService.cs:61` |
| 30 | **`FailedWriteCount` metric exposed "for future Health Monitoring" but never consumed** — dangling metric. | `ISiteEventLogger.cs:32-40` |
| 31 | **`StateTransitionValidator` allows Delete from `NotDeployed`; spec matrix says No** (deliberate per code comment, contradicts doc). | `StateTransitionValidator.cs:38-39` |

---

## Tier 3 — Intentional deferrals (correctly documented — NOT defects)

Knowingly punted, with extensible seams and explicit doc notes. `[PERM]` = permanent / v-next; `[SLICE]` = deferred-to-a-later-slice with seam present.

**Centralized Audit Log (#23)**
- `[PERM]` Hash-chain tamper evidence (v1.x). `verify-chain` CLI is a no-op stub that prints "not enabled in this release". — `AuditCommands.cs:243-246`; `AuditVerifyChainHelpers.cs:6-8`
- `[PERM]` Parquet export/archival. Server returns HTTP `501`; CSV + JSONL implemented. — `AuditEndpoints.cs:188-194`; `AuditExportHelpers.cs:139-148`
- `[PERM]` Per-channel retention overrides. — `2026-05-20-audit-log-code-roadmap.md:16`
- `[PERM]` Tag-cascade for `ParentExecutionId` — only the inbound-API→routed-site bridge is built; trigger-driven runs pass `parentExecutionId = null`. — `ScriptActor.cs:404,429`; `2026-05-21-audit-parent-executionid-design.md:209`
- `[PERM]` ExecutionId/ParentExecutionId backfill on historical rows; SourceNode backfill on legacy rows; per-node stuck-count KPIs.
- `[PERM]` Structured/response-header response capture; inbound request-header capture; per-method opt-out; `AuditInboundCeilingHits` metric. — `2026-05-23-inbound-api-full-response-audit-design.md:113-127`
- *Uncertain:* CLI `audit tree` command (doc "maybe", not found in CLI).

**Notifications (#8 / #21)**
- `[SLICE]` Teams (and all non-Email) notification types — `INotificationDeliveryAdapter` seam exists, only `EmailNotificationDeliveryAdapter` implemented; `NotificationType` enum is Email-only. Missing-adapter path parks gracefully. — `NotificationType.cs:6-9`; `NotificationOutboxActor.cs:457-474`
- `[SLICE]` Central UI notification-list form has no `Type` selector (Email hard-coded). — `NotificationListForm.razor`
- `[PERM]` Historical/trend KPI charts (no time-series store).

**Native Alarms / MxGateway / OPC UA**
- `[PERM]` Native-alarm ack/shelve/suppress write-back; central alarm tables/history/journal; alarm-driven notifications/scripts — read-only by design. — `2026-05-29-native-alarms-design.md:201-206`
- `[SLICE]` Dedicated operator Alarm Summary page (DebugView only for now).
- `[PERM]` MxGateway secured writes (operator+verifier).
- `[SLICE]` OPC UA address-space search; `BrowseNext` paging. — `RealOpcUaClient.cs:574`
- `[PERM]` OPC UA type-info surfacing; bulk override import/CSV.
- `[SLICE]` OPC UA "Verify endpoint" connectivity button; cert-management UI.

**Transport (#24)**
- `[PERM]` Site-scoped / instance-scoped artifact transport (needs name-mapping subsystem).
- `[PERM]` Direct cluster-to-cluster pull; asymmetric bundle signing; differential/incremental bundles.
- `[PERM/SLICE]` Per-line/Myers diff for Modified artifacts (coarse line-count delta only). — `ArtifactDiff.cs:18-25`

**TreeView**
- `[SLICE/PERM]` R6 lazy-loading, R7 keyboard nav, R16 multi-select — spec marks all "(Deferred)". — `Component-TreeView.md:87-93,288-295`

**Templates / Data Connections / Triggers UI**
- `[SLICE]` Template tree search/filter; `[PERM]` folder drag-drop, sibling reorder, root context menu.
- `[PERM]` Move data connection between sites; `[SLICE]` connection live-status indicators (blocked on DCL state surfacing).
- `[SLICE]` Base-template versioning "update-derived" flow; multiple inheritance levels; `[PERM]` promote-derived-to-base, cross-tenant libraries.
- `[SLICE]` Strict expression-trigger analysis kind; `[PERM]` WhileTrue trigger mode for alarms.
- `[SLICE]` Schema-driven value-entry forms; schema hover/completion; `[PERM]` JSON Schema `$ref` reuse / template-level schema library.

**Cached-call tracking (#6 / #22)**
- `[SLICE]` CLI surface for site-local Retry/Discard of cached calls; `[PERM]` unified notifications+site-calls outbox page.

**UI audit backlog (`2026-05-12-ui-audit.md:536-554`)**
- `IDialogService` modal abstraction; design-tokens/CSS-vars; dark-mode/theming; shared pagination+filter component; accessibility pass; replacing SignalR debug-view streaming.

**Environment / tooling**
- `[PERM]` True air-gapped second environment (env2 shares MSSQL/LDAP/SMTP); 3rd/4th env; `--env` flag on `deploy.sh`.
- `[PERM]` Repo/folder rename (kept as ScadaBridge to preserve context).
- `[SLICE]` Playwright alarm-override UI coverage.

---

## Tier 4 — Doc↔code drift (code is fine; docs describe a superseded architecture)

Worth fixing for anyone relying on the docs as the spec.

**Config DB / Commons re-architecture not reflected in specs (High doc-impact):**
- `AuditLog` table collapsed to 10 canonical + `DetailsJson` + 6 PERSISTED `JSON_VALUE` computed cols; doc still lists ~24 typed columns (`Kind`, `HttpStatus`, `RequestSummary`, …). — migration `20260602174346_CollapseAuditLogToCanonical.cs`; `Entities/AuditLogRow.cs:54-136`
- `AuditEvent` moved out of Commons into the external `ZB.MOM.WW.Audit` NuGet package; doc (REQ-COM-1/3/5b) still describes it as a Commons type. — `Commons.csproj:11`
- `ApiKey` entity / API-key persistence retired to shared `ZB.MOM.WW.Auth.ApiKeys` SQLite store; doc still lists `ApiKey` + `ApprovedApiKeyIds`. — migration `20260602092753_RetireInboundApiKeyStore.cs`

**CLI docs drift (README is the stale doc; `Component-CLI.md` mostly matches code):**
- Entire `bundle` (Transport #24) command group is shipped + registered but documented in **neither** `Component-CLI.md` **nor** `CLI/README.md`. — `Program.cs:36`; `BundleCommands.cs:24-372`
- `security api-key create` requires undocumented `--methods` (Required); docs show only `--name`. — `SecurityCommands.cs:41-45`
- `security api-key update`/`delete` use `--key-id`; docs document `--id` (and an unwired `--name` on update). — `SecurityCommands.cs:60,71`
- `security api-key set-methods` subcommand exists in code, documented nowhere. — `SecurityCommands.cs:91-102`
- `api-method create` uses required `--script`; docs document `--code` + `--description` (neither exists). README is internally inconsistent (create=`--code`, update=`--script`). — `ApiMethodCommands.cs:57-62`
- `db-connection create`/`update` documented with `--provider`; code has no such option. — `DbConnectionCommands.cs:56-72`
- Widespread README option-name drift where `Component-CLI.md` already matches code (scope-rule `--mapping-id`, health `--site`/`--keyword`, template attribute `--value`/`--data-source`, template alarm `--trigger-type`/`--priority`/`--trigger-config`, composition delete `--id`, etc.).
- `audit query` doc lists `--page` (code is keyset-only `--all`); undocumented `--execution-id`/`--parent-execution-id` filters exist.

**Stale "deferred" markers for things that have actually SHIPPED:**
- Transport CLI (`bundle export/preview/import`) — design doc §13 said "deferred"; now implemented.
- `SourceNode` capture — `.tasks.json` shows all 21 tasks "pending"; fully implemented across Commons/AuditLog/NotificationOutbox/SiteCallOperational.
- Site Call Audit Retry/Discard relay — DI comment says deferred; implemented + wired (`SiteCallAuditActor.cs:150-156,450-505`; `AkkaHostedService.cs:580-589`).
- Bundle-import audit filter UI (Transport-012) — doc says deferred follow-up; shipped (`ConfigurationAuditLog.razor` `?bundleImportId=` filter).
- Redaction/payload-cap "deferred to M5" comments in Site Runtime — already shipped (`ScadaBridgeAuditRedactor`, `AuditLogOptions.DefaultCapBytes/ErrorCapBytes`).
- `AuditLogPage.HandleRowSelected` class comment says "no-op seam"; method is fully wired (opens drawer).

**Other doc/spec inconsistencies (code richer/different than doc):**
- Security role names: doc says Admin/Design/Deployment; code uses Administrator/Designer/Deployer/Viewer (canonicalized via migration).
- `SiteCall` entity field names diverge from doc (`Channel` not `Kind`, `SourceSite` not `SourceSiteId`, adds `HttpStatus`/`IngestedAtUtc`).
- `ExecuteReader` audited as `DbWrite` (read/write distinguished via `Extra` JSON `op`, not a distinct `AuditKind`).
- Inbound audit doc references `ApiInbound.Completed`; actual kinds are `InboundRequest`/`InboundAuthFailure`.
- `Teams` claimed present in `NotificationType` enum by Commons/ConfigDB docs; enum is Email-only.
- Commons under-documents shipped code: MxGateway endpoint serializer/validator/config, `Observability/ScadaBridgeTelemetry.cs`, `IInboundApiKeyAdmin`, `IAuditActorAccessor` — none in the doc folder map.
- `IHealthMonitoringRepository` listed in ConfigDB repo table but doesn't exist (doc annotated "future").
- `requirements-traceability.md` and many `.md.tasks.json` show "Pending" for shipped features — they track *plan generation*, not implementation; unreliable as a status source.
- `ExternalSystemForm` "Recent audit activity" drill-in omits `channel=ApiOutbound` and uses exact-match `target` instead of starts-with (sibling `ApiKeyForm` link is correct). — `ExternalSystemForm.razor:20-24`

---

## Code-level sweep — investigated and ruled out (false positives)

For completeness, items that *look* unfinished but are intentional:
- ~44 empty `catch` blocks — all have explanatory comments / intentional fallback (JSON parse → default; disposal-race `ObjectDisposedException`). None silently swallow real errors.
- `SiteNotificationRepository` / `SiteExternalSystemRepository` write methods throw `NotSupportedException` — by design (site config is read-only, managed via central deployment).
- `StubOpcUaClient` (canned data; `BrowseChildrenAsync` throws `NotImplementedException`) — dev/test-only; production wires `RealOpcUaClientFactory`/`RealMxGatewayClientFactory` (`DataConnectionFactory.cs:38-47`).
- `NoOpSiteStreamAuditClient`, `SandboxNotifyHelper`, sandbox host fakes — legitimate DI-default / test composition seams.
- `AddSecurityActors` / `AddTemplateEngineActors` "Phase 0 placeholder" registrations — intentional empty seams (actor wiring lives in Host).
- Migration `Down()` `NotSupportedException`, MxGateway/Bundle version-rejection `NotSupportedException`, `AuditWriteMiddleware` write-only-stream `NotSupportedException` — intentional guards.
- Management Service: 113 handlers; all wire-registered `Mgmt*` commands dispatched. The three "unhandled" (`ResolveRolesCommand` retired; `BrowseNodeCommand`/`ReadTagValuesCommand` routed direct-to-site) are intentional.
- Central UI: no stub/placeholder pages, no `NotImplementedException`, no "coming soon" banners, no no-op `@onclick`. `disabled=`/`placeholder=` usages are legitimate (loading guards, edit locks, HTML hints).

---

## Phase completeness (self-reported)

All 11 phases report Complete with passing verification gates:

- **Phase 0** Solution Skeleton — Complete (gate 11/11, 57 tests).
- **Phase 1** Central Foundations — Complete (gate 20/20, 186 pass + 1 live-LDAP skip).
- **Phase 2** Modeling & Validation — Complete (gate 9/9, 359 tests).
- **Phase 3A** Runtime Foundation — Complete (gate 13/13, 389/389).
- **Phase 3B** Site I/O & Observability — Complete (gate 11/11, 541 cumulative).
- **Phase 3C** Deployment & Store-Forward — Complete (terse checklist).
- **Phase 4** Operator UI — Complete (terse checklist).
- **Phase 5** Authoring UI — Complete (terse checklist).
- **Phase 6** Deployment & Ops UI — Complete (terse checklist; Codex external-review step skipped, best-effort).
- **Phase 7** Integrations — Complete (terse checklist; Q12 SMTP-OAuth2 is a test-env dependency).
- **Phase 8** Production Readiness — Complete (terse checklist).

The ~665 unchecked `- [ ]` items in phase *plan* docs are spec-traceability references (each dispositioned Pass / Out-of-scope in Forward/Reverse tables), a documentation style — not a TODO list.

**Operational (not code):** `docs/deployment/production-checklist.md` has ~60 unchecked install-time operator steps (env vars, connection strings, firewall ports 8081/636/587/1433, TLS certs, smoke tests).

---

## Confidence & caveats

- **High confidence** on Tier 1 — each item verified by reading the code (class/interface existence + absence of callers via grep); top items corroborated by 2+ independent agents.
- Terse Phase 3C–8 checklists self-report "Complete / tests passing" with no per-gate breakdown; test counts for those phases were not independently re-run.
- Actual `src/` artifacts were treated as truth over `.tasks.json` status fields, which are demonstrably stale.
- Items marked *Uncertain* (e.g. `audit tree` CLI, per-channel retention) rest on doc text only.

## Recommended next steps

1. **Wire the two never-started audit actors** (#3, #4) — highest impact, smallest blast radius (DI/Host wiring + `IPullAuditEventsClient` impl).
2. **Site Call Audit reconciliation + purge** (#5) — same shape as #3/#4.
3. **Decide on script compilation/security** (#1, #2) — either implement the Roslyn gate or downgrade the spec's claims; currently the strongest functional + security gap.
4. **Site Event Logging categories** (#6) — inject `ISiteEventLogger` into the 5 missing subsystems.
5. **Reconcile Tier-4 doc drift** — update Config DB / Commons specs for the audit/auth re-architecture and the CLI docs for the `bundle` group + option names.