dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0
..
pull from: dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0

126 Commits
fix/cached-telemetry-drain-hot-loop .. ui/central-ui-cleanup-sweep

Author SHA1 Message Date
Joseph Doherty d14e0ee4b1 fix(ui): gate detail modals on user intent, not on the row resolving 
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).

The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.

Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.

Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.

The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
 2026-08-11 06:03:06 -04:00
Joseph Doherty 9e243493fb ui: Central UI density/consistency sweep + Theme 0.4.1 
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.

The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.

Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.

NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.

Build 0/0; suite green apart from that one intermittent failure.
 2026-08-11 05:50:12 -04:00
Joseph Doherty b6f383a225 docs(notifications): record EWS live-gate PASS (rig -> on-prem Exchange, Delivered first attempt) 2026-08-10 07:29:06 -04:00
Joseph Doherty ba994a59c5 docs: final-review nits — test-stub wording, README tech-stack row, UI-audit follow-up 2026-08-10 07:07:00 -04:00
Joseph Doherty 05d0631cdd chore(plans): mark EWS transport plan tasks 10-11 complete 2026-08-10 07:00:12 -04:00
Joseph Doherty 00d8a923af docs(notifications): EWS transport docs; close Q12 as superseded; design-doc corrections from execution reviews 2026-08-10 06:56:56 -04:00
Joseph Doherty 82f52e81ce fix(ui): mirror the EWS username:password credentials rule on the SMTP page 2026-08-10 06:53:15 -04:00
Joseph Doherty 3e490f2a7d test(ui): bUnit coverage for the SMTP page transport selector state machine 2026-08-10 06:50:39 -04:00
Joseph Doherty acbb9eafa5 fix(management): EWS write gate requires username:password credentials 2026-08-10 06:48:15 -04:00
Joseph Doherty 8df15b34b8 fix(transport): carry Transport + OAuth2 authority/scope on SmtpConfigDto (OAuth2 fields were silently dropped) 2026-08-10 06:43:43 -04:00
Joseph Doherty 2f217f5742 docs(cli): document notification smtp update --transport 2026-08-10 06:39:49 -04:00
Joseph Doherty 6f8b9c755d feat(ui): EWS transport selector on /notifications/smtp 2026-08-10 06:38:46 -04:00
Joseph Doherty 0fe1972960 feat(cli): notification smtp update --transport smtp|ews 2026-08-10 06:35:33 -04:00
Joseph Doherty 4e633b1e64 feat(management): Transport on UpdateSmtpConfigCommand with EWS shape validation 2026-08-10 06:33:29 -04:00
Joseph Doherty 8657fae14f fix(notifications): EWS sender review follow-ups — https guard, transient-path logging, test hardening 2026-08-10 06:29:37 -04:00
Joseph Doherty 08957cc907 feat(notifications): EWS branch in the email delivery adapter 2026-08-10 06:27:19 -04:00
Joseph Doherty 2ee5586406 test(notifications): correct XXE guard comment — LINQ-to-XML does not prohibit DTDs by default 2026-08-10 06:21:19 -04:00
Joseph Doherty 6864890e5f fix(notifications): prohibit DTD processing in EwsResponseParser (XXE guard) 
The review asked for a regression test pinning DTD-prohibited parsing. Writing
it showed the premise was wrong: XDocument.Parse permits an internal DTD subset
and expands its entities, so a DOCTYPE-bearing response body parsed fine and the
guard did not exist (entity-expansion DoS on external input).

Parse now goes through XmlReader with DtdProcessing.Prohibit and a null
XmlResolver. The regression test feeds a DOCTYPE + ENTITY payload shaped as a
well-formed EWS error response, so it fails if DtdProcessing is ever loosened
rather than passing for the unrelated-XML reason.
 2026-08-10 06:20:48 -04:00
Joseph Doherty abc58e6394 feat(notifications): no-SDK EWS SOAP mail sender with typed transient/permanent classification 2026-08-10 06:18:08 -04:00
Joseph Doherty abb1581d45 feat(notifications): EWS CreateItem envelope builder + response parser 2026-08-10 06:12:46 -04:00
Joseph Doherty 3da84825fc feat(notifications): additive Transport column on SmtpConfigurations 2026-08-10 06:09:12 -04:00
Joseph Doherty d21b7a5a48 feat(notifications): EmailTransport enum + parser for the EWS transport discriminator 2026-08-10 06:07:14 -04:00
Joseph Doherty 9f4d7d4bcb docs(notifications): EWS email transport implementation plan (11 tasks) 2026-08-10 06:05:04 -04:00
Joseph Doherty b2ea9c6c74 docs(notifications): design EWS email transport for the outbox; close Q12 as superseded 
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
 2026-08-10 05:56:41 -04:00
Joseph Doherty e9c412e528 fix(host): unhandled boot exception now kills the process instead of wedging the container (#34) 
Root cause: dotnet runs as container PID 1 and Linux ignores default-action
signals sent to PID 1, so the runtime's unhandled-exception path (banner,
then abort() -> SIGABRT) could never terminate the process — it printed the
trace and spun the main thread at 100% CPU with the container `running`,
so `restart: unless-stopped` never fired. Reproduced deterministically:
same StartupValidator throw exits 134 under an init process and wedges
without one.

Two layers, each covering the other's gap:
- Program.cs registers an AppDomain.UnhandledException handler before the
  first statement that can throw: prints the trace, best-effort flushes
  Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform,
  134 preserves the 128+SIGABRT crash code, and it covers every thread,
  not just the boot window. It cannot fire under WebApplicationFactory
  (the test host catches entry-point exceptions), so the designed
  boot-refusal exceptions still propagate to tests unchanged.
- docker-compose: init: true on all 8 nodes for the crash paths that
  bypass the managed event (Environment.FailFast, runtime-internal aborts).

The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out
(exactly one call, only inside the handler); Environment.Exit still fires
the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the
crash path skips nothing abort() kept. New pin test keeps the handler ahead
of the configuration build.

Live-verified on the rig image: crash now yields Exited (134) +
RestartCount climbing under `unless-stopped`, trace intact, with and
without init; full 8-node rig redeployed healthy with docker-init as PID 1.

Closes #34.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-08 05:23:26 -04:00
Joseph Doherty e697477c1f feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator 
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 11:39:28 -04:00
Joseph Doherty 4d7f09d550 test(secrets): central-shared-store live gate 5/5 — shared SQL store + hub failover proven on the rig 
Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 11:28:58 -04:00
Joseph Doherty 68f812eaa4 feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.0 and wire hub fallback endpoints 
Pin all five ZB.MOM.WW.Secrets* packages 0.4.1 -> 0.5.0, which brings
SecretsGrpcHubClientOptions.FallbackEndpoints and the package's internal
FailoverSecretsHubReader. A site whose GrpcHub section lists fallback
endpoints now fails a sweep over to the next central instead of stalling
on a downed primary - safe ONLY because both central nodes serve one
shared SQL secret store (scadaproj#4), so either hub answers with the
same manifest; the appsettings comments say so and warn against listing
endpoints backed by independent stores.

appsettings.json gains "FallbackEndpoints": [] with a _fallbackEndpoints
comment, and the _endpoint note's single-endpoint-stall caveat is scoped
to the empty-list case it now only applies to.

Wiring pins (red first on 0.4.1): site + Grpc + one fallback resolves
ISecretsHubReader to FailoverSecretsHubReader with the
"zb-secrets-grpc-hub:fallback:0" keyed channel present; zero fallbacks
keeps the plain GrpcSecretsHubClient and no fallback channel - the
pre-0.5.0 container shape byte-identical.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:55:38 -04:00
Joseph Doherty aebdd56b52 docs(secrets): review follow-ups — pre-Serilog window note, ThrowIfNull hygiene, honest site-purity scan bound 
The expander comment now records that a SQL outage at central boot exits
pre-Serilog with a bare stderr trace (honest, restart-retryable, unenriched)
and why the role-reading predicate deliberately stays in Program.cs rather
than SecretsRegistration (that class refuses config-read roles by design).
EnsureCentralSharedStoreConnectionString gains the file's standard
ThrowIfNull. The site-purity scan's doc no longer overclaims: two
factory-lambda descriptors evade it individually; the pin holds because
three concrete-type registrations from the same call cannot.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:47:10 -04:00
Joseph Doherty a358244d9e fix(secrets): share the central connstr validation with the Layer-A expander 
The blank/${secret:} pre-checks lived only in AddScadaBridgeSecrets, but on
a real central boot the Layer-A expander runs FIRST and would hand a bad
value to SqlConnection, burying the designed message under a generic
'initialization string' format error. Extracted both checks into
EnsureCentralSharedStoreConnectionString — one definition, called by the
expander (earliest point) and by registration (covers embedded/test
composition) — same single-source lesson as the UsesGrpcHub predicate.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:36:13 -04:00
Joseph Doherty 43e87a7492 feat(secrets): central's Grpc-mode store is the SHARED SQL-Server store (scadaproj#4) 
In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.

- SecretsRegistration: two fail-closed pre-checks before any registration on
  the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
  naming the key (an independent store per central node is the recorded
  defect), and a value containing ${secret: throws naming the bootstrap
  circularity (the expander needs this store to resolve references). Site
  registrations are byte-identical to before; SqlServer mode and
  replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
  non-blank connection string migrates and resolves pre-host ${secret:}
  references through the shared SQL store, so expanded values can never
  diverge from what the running node serves. Every other case keeps the
  SQLite path unchanged; blank-connstr central deliberately falls through so
  the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
  central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
  ${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
  descriptor sweep), central fixtures carry the now-required connstr.

Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:33:28 -04:00
Joseph Doherty f6c3f7c593 test(secrets): live gate 4/4 — check 4 re-run and PASSES on 0.4.1 
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.

The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.

Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.

Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.

The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.

Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.

4/4. Merging.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:51:39 -04:00
Joseph Doherty 6d38e89be0 deps(secrets): 0.4.1 — hub denial warning (live-gate check 4) 
The 2026-08-07 live gate's check 4 failed one clause of three: both auth
negatives were denied with a byte-identical Unauthenticated status and no token
or plaintext reached any log, but the hub recorded a denial only at Information
level, via Grpc.AspNetCore.Server rather than the gate itself. A follower whose
token was mis-rotated would therefore stop converging while central showed
nothing above INF.

That was a property of the library, not of this branch, so it was fixed there and
shipped as 0.4.1 (scadaproj main c86cead): SecretsHubAuthInterceptor now emits a
rate-limited Warning summarising each denial window, breaking the count down by
cause, with the first denial after startup or a quiet window warning immediately
so a single probe is never silent. Wire behaviour is untouched - the denial is
still one uniform Unauthenticated with one detail, and still carries no token
material - so this is additive on the server's diagnostics only and nothing a
follower observes changes.

All five ZB.MOM.WW.Secrets* pins move together. Splitting them is not an option
worth having: Abstractions carries the StoredSecret shape the Grpc wire mirror is
written against, so a mixed set is a silent structural mismatch rather than a
build error.

Build clean at 0 warnings; the secrets wiring + hub-mapping pins are 31/31.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:36:57 -04:00
Joseph Doherty 9d5cf7100e test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged 
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:01:55 -04:00
Joseph Doherty fc784b4137 fix(secrets): registration and hub mapping share the UsesGrpcHub predicate (review Important) 
AddScadaBridgeSecrets now branches on UsesGrpcHub instead of re-deriving the
same condition, so the doc's single-predicate claim is enforced rather than
aspirational; the role split is a switch with a default-throw so a future
third role must choose its hub half explicitly.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:33:47 -04:00
Joseph Doherty c8e90daafb test(secrets): pin both hub halves, the mode key, and the mapping gate 
Extends the existing build-and-resolve wiring suite rather than asserting over
ServiceDescriptors, for the reason that file already documents: a registration
can look correct as a descriptor list and still fail on first resolve.

Role separation is pinned in both directions — central registers the hub's
fail-closed interceptor and NOT the sweep, a site registers the sweep and NOT
the hub — because only one of those is a security failure and testing the
happy half would not catch it. The sweep is asserted by resolving IHostedService,
which builds its whole graph (reader, keyed channel, local store) and is where
a forgotten AddZbSecrets would surface; GrpcSecretSyncService is internal to
the package, so it is matched by assembly + type name the way the SqlServer
replicator's services already are.

SqlServer mode gets regression pins with the mode key both unset and named
explicitly, plus one asserting it stays role-agnostic — both nodes sync
bidirectionally against the same database, and the role parameter added for the
hub must not have quietly changed that.

The mapping tests assert over the app's real endpoint data sources, on the WIRE
route (/zb.mom.ww.secrets.hub.v1.SecretsHub/...) rather than the C# type, since
the route is what a follower addresses. One of them pins exactly the two READ
methods: pull-only is a property of the contract, and this is where a future
package version growing a write RPC would become visible instead of silently
opening a path for a site to overwrite central.

Numeric mode values get their own test. Enum.TryParse accepts any integer,
including ones outside the enum, so "7" would otherwise select a mode that does
not exist and fall through to the SqlServer branch.

Verified red-first by mutation on the finished implementation: swapping the two
role branches reds 8 (both role pins, both fail-closed pins, both mapping
pins); deleting the UsesGrpcHub check in the map extension reds exactly the two
"maps no hub endpoint" cases — the unauthenticated-hub scenario; dropping
Enum.IsDefined and letting UsesGrpcHub ignore Enabled reds the out-of-range
value and the flag-off-with-full-hub-config cases. 31/31 green restored.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 127ec25425 feat(secrets): wire the pull-only gRPC secrets hub — central hosts, sites sweep 
scadaproj#3: the production secrets topology is central hosting a pull-only
gRPC hub with site nodes sweeping it. The SqlServer replicator stays in the
codebase and keeps working exactly as it did, but it is not the production
path — it needs every site node to hold a connection string to central's
database, which breaks ScadaBridge's standing rule that sites talk to central,
not to central's DB.

Selection is a new Secrets:Replication:Mode key alongside the existing
Secrets:Replication:Enabled flag. Absent or blank means SqlServer, so an
existing configuration that sets only Enabled behaves identically; an
unrecognised value is refused at startup naming the key, and refused whenever
it is present rather than only when replication is on — a typo should fail the
boot that introduced it, not some later boot that flips an unrelated flag.

Which HALF a node composes is a parameter, not a config key. Both composition
roots already know statically which they are (Program.cs is central,
SiteServiceRegistration is a site), and a role read from configuration is a
role that can be got wrong in the one direction that matters: a site hosting
the hub would serve central's whole secret inventory from inside the site
network to anything holding the shared token.

The hub is mapped onto the EXISTING central h2c control-plane listener
(ScadaBridge:Node:CentralGrpcPort, default 8083) beside CentralControlService
— the same listener and the same addressing convention sites already use, and
the same shape as the site's LocalDb sync endpoint sharing its gRPC port: two
disjoint service prefixes, two independent fail-closed gates. The
CentralControlAuthInterceptor on AddGrpc is prefix-scoped and passes hub calls
through; the hub's own SecretsHubAuthInterceptor, attached per-service by
AddZbSecretsGrpcHub, gates them on Secrets:GrpcHub:BearerToken.

Registration and mapping share one predicate (UsesGrpcHub) deliberately.
Mapping without registering would map a hub whose interceptor was never
attached — an anonymous endpoint serving every secret central holds, on a node
that looks completely healthy — so the two must not be able to drift.

gRPC mode fails CLOSED where SqlServer mode degrades: a missing endpoint or
bearer token is a startup failure, not a warning plus a local-only store. The
degraded outcome is precisely what the hub exists to prevent (a site quietly
serving secrets that never converge), and the package's own errors name the
exact key, so they are left unwrapped.

Templates are default-OFF: Enabled stays false, Mode stays SqlServer, and
Secrets:GrpcHub ships with an empty BearerToken and Endpoint. Empty is
fail-closed, not open. The token is documented as appsettings/env only — never
a ${secret:} reference, since resolving one is what the hub exists to make
possible — the same bootstrap rule the mesh pre-shared keys follow.

Known asymmetry, recorded in the template: CentralGrpcEndpoints is a LIST that
fails over across the central pair, but the hub client dials a SINGLE endpoint,
so a sweep against a downed central node stalls instead of failing over. That
is survivable — the sweep is best-effort and the site keeps serving its full
local last-known-good store — but secrets stop converging until that node is
back.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 57f08c213b deps(secrets): bump ZB.MOM.WW.Secrets* to 0.4.0 and take the Grpc replicator 
0.4.0 adds ZB.MOM.WW.Secrets.Replicator.Grpc — the pull-only central secrets
hub scadaproj#3 selected as ScadaBridge's production topology. The four
existing pins move with it rather than straddling two versions: they share the
Abstractions surface the replicators bind to, and a mixed set is a restore that
resolves but composes types from two different builds of the same seam.

One package carries both halves (server + sweep client); which half a node
composes is a registration-time decision, so only the Host — the composition
root for both roles — references it.

nuget.config already maps ZB.MOM.WW.Secrets.* to the dohertj2-gitea feed, so
no source-mapping change was needed. Restore verified against the feed.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:17 -04:00
Joseph Doherty 4df3a55824 docs: truth sweep — retire stale registers, reconcile ledgers with shipped state 
- deferred.md: DELETED (git rm) — stale 2026-07-10 duplicate of the canonical
  deferred-work register; this completes archreview R2-08 T11 (the file was
  tracked, not untracked as the task assumed)
- ScadaBridge-docs-issues.md, ScadaBridge-docs-fixed.md: DELETED (git rm) —
  generated 2026-07-10 CommentChecker reports, already consumed; completes
  R2-08 T13 (also tracked, not untracked)
- stillpending.md: prepended historical-snapshot banner (2026-06-15 audit;
  Tier-1 table is not current open work)
- docs/plans/phase-8-checklist.md: replaced the unevidenced 'Complete / All
  passing' stub with the honest state per register row 25 (WP-4 target-scale
  load test never run)
- archreview/plans/00-MASTER-TRACKER.md: R2-01 T2 live failover drill
  annotated RESOLVED 2026-08-01 (PLAN-R2-01 T4 + docker/failover-drill.sh +
  SbrFailoverTests); R2-08 T11/T13 recorded completed by this sweep
- docs/plans/2026-07-22-clusterclient-to-grpc-plan.md: P3 deferred-RPCs note
  updated (all four live-proven 2026-08-01, 1c99d6fa); ClusterClientSiteAuditClient
  naming follow-up marked DONE (63c16d69)
- docs/plans/2026-05-28-opcua-tag-browser.md.tasks.json: Task 19 flipped to
  completed (manual smoke PASS 2026-08-01, 6dc5d94c)
- archreview/plans/PLAN-R2-0[1-8]*.tasks.json: all-pending manifests reconciled
  with the authoritative tracker (round 2 merged @ 1930f19b) — flipped to
  completed except R2-08 T1/T2 which remain pending needs-user
- docs/operations/2026-07-16-secrets-clustered-master-key.md: correction banner
  (SQL-hub replication shipped 8e12f994; KEK-rotation + clustered-secrets
  runbooks ship with ZB.MOM.WW.Secrets)
- docs/plans/2026-07-19-localdb-phase2-live-gate.md: external-system-delete
  observation annotated RESOLVED (2d03f2d5 reconciles deletions incl.
  external_systems)

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 01:56:33 -04:00
Joseph Doherty 7caa8bfd99 fix(localdb): disable SQLite pooling on legacy reads - a Windows site node could not boot 
SiteLocalDbLegacyMigrator opens each pre-Phase-2 file read-only, copies its rows into
the consolidated site database, then renames the file so a later boot skips it.
Microsoft.Data.Sqlite POOLS connections, so disposing one returns it to the pool and
leaves the underlying sqlite3 handle - and the OS file handle - open. On Windows the
subsequent File.Move throws IOException ("being used by another process"), the
migration faults out of AddZbLocalDb's factory, and the site node does not start at
all. There is no partial-migration path: it is a hard boot failure, once, on the first
upgrade past LocalDb Phase 2.

Why it shipped: POSIX rename ignores open handles, so this cannot reproduce on Linux or
macOS. Every existing rename assertion in SiteLocalDbLegacyMigratorTests - including
LegacyTrackingRows_AreCopiedAndTheFileIsRenamed - passes with the bug fully present, and
the docker rig migrates cleanly. It was found by pre-flighting the wonder-app-vd03
upgrade against COPIES of that box's real site databases; an earlier pre-flight pass
with empty data directories had nothing to drain and passed clean.

Fix: both legacy read-only connection strings now go through one
LegacyReadOnlyConnectionString helper carrying Pooling=False.

The accompanying test is deliberately white-box. A behavioural assertion cannot
discriminate here on the platform this suite runs on, so it pins the connection string
instead; it is red without the fix.

Verified: Host.Tests 440/440, 0 warnings. The build deployed to wonder-app-vd03 carries
this change (as the then-uncommitted fix) and has been running since 2026-08-05.
 2026-08-05 17:06:54 -04:00
Joseph Doherty b8f91bab2d feat(cluster): enable the #33 bootstrap guard on the docker rig — live gate PASS 
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
 2026-08-02 01:04:55 -04:00
Joseph Doherty 3c9b101dfc docs(alarms): document the Alarms accessor + AckTime, tick MES plan Phase 1 
Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:

  - Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
    it is not scope-prefixed, why it is read-only, why placeholder rows are
    included), the full ScriptAlarm shape, AckTime on the enriched
    AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
    persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
    builds its CDC triggers from the column list at registration time).
  - Component-DataConnectionLayer.md already carried the AckTime section in the
    first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
  - Component-ScriptAnalysis.md: accessors returning domain types return the
    SAME type on both surfaces, and the trust-model note that a deny-list needs
    no entry for a new globals member -- only that its return type resolves in
    a permitted namespace.
  - Component-InboundAPI.md records the NEGATIVE decision: there is
    deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
    per-instance and lives on the site's Instance Actor, so the read goes
    through a routed site script and the filtering happens where the data is;
    central stays a thin router.
  - CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
  - The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
    what was actually built (incl. the two choices that differ from the plan's
    "or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
    and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
    stay open -- they are deployed config and need a live rig.
 2026-08-01 13:12:46 -04:00
Joseph Doherty d0af884760 feat(scripts): add the Alarms.CurrentAsync() read accessor for site scripts 
MES alarm-status API §5.2 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 tasks 2-4). Site `Call` scripts had NO way to read alarm condition
state: the `Alarm` global exists only inside an on-trigger handler and
describes the one alarm that fired, and native mirrored conditions were
reachable only from the Debug View. That gap blocked the CvdReactor
SimpleAlarmStatus/AlarmStatus scripts entirely -- they cannot be written
without it. `Alarms.CurrentAsync()` closes it.

The data was already local: the script runs inside its own Instance Actor's
context, so this is a LOCAL Ask -- the same mechanism attribute reads use, no
cross-cluster hop. A dedicated GetAlarmSnapshotRequest/Response is used rather
than reusing DebugSnapshotRequest, which would materialise every attribute
value on every alarm poll; both are served from the same
BuildAlarmStatesSnapshot(), so the script view and the operator's Debug View
can never disagree.

Deliberate shape decisions:
  - NOT scope-prefixed, unlike Attributes. Alarm identity is not a
    scope-relative attribute name (computed alarms are keyed by configured
    name, native conditions by a source-supplied reference), so prefixing
    would hand a composed script a silently truncated list.
  - Read-only. Native alarms are a read-only mirror of the source (no
    ack-back), so no acknowledge/shelve operation is exposed.
  - Placeholder rows are NOT pre-filtered: a caller must be able to tell
    "binding configured and quiet" from "binding unknown". The documented
    filter is `Active && !IsConfiguredPlaceholder`.
  - ScriptAlarm lives in Commons so the runtime accessor and the compile-only
    surface project to the SAME type -- a script binding at the design-time
    gate binds identically at the site. Condition is the authority for
    active/acked/severity, so one filter expression works across computed and
    native alarms.

Mirrored on BOTH design-time surfaces. ScriptCompileSurface is covered by the
reflection parity guard (AlarmsAccessor added to its mirror pairs). The Central
UI Test-Run SandboxScriptHost is the third, hand-maintained mirror that the
parity test cannot reach (Central UI does not reference Site Runtime); without
it the design page would false-flag CS1061 on scripts the deploy gate accepts.
It throws a labelled ScriptSandboxException at run time -- there is no central
route to per-instance alarm state, and returning an empty list would read as
"nothing is in alarm", which is worse than an error.

ScriptTrustPolicy needs NO change, and the reason is structural rather than
incidental: the trust boundary is a deny-list over API roots, not an allow-list
of context members. A test pins that no ForbiddenScopes entry prefixes the
Commons script-surface namespace, so a future deny-list entry cannot silently
make ScriptAlarm untouchable.

Tests: 6 accessor cases (Ask contract, full native projection incl. AckTime,
unacked, computed-alarm derivation, placeholder visibility, scope-independence),
2 InstanceActor snapshot cases incl. equality with the Debug View row set, the
full MES script shape compiling against ScriptCompileSurface, 2 trust cases,
and a sandbox diagnose-clean case reading every projected ScriptAlarm field.
 2026-08-01 13:12:30 -04:00
Joseph Doherty 01bcca992c feat(alarms): thread an additive AckTime through the native-alarm mirror 
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.

Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.

Provenance is honest, never fabricated:
  - OPC UA A&C supplies a TRUE ack instant, so we now select it:
    AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
    index 18, APPENDED so the positional reads at 0-17 keep their meaning.
    Servers that omit the field fall back to the event's own Time.
  - MxAccess Gateway supplies none, so the ack transition's own timestamp is
    used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
    re-subscribe snapshot restores one from LastTransitionTimestamp rather
    than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.

Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.

Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.

Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
 2026-08-01 13:12:04 -04:00
Joseph Doherty 6dc5d94cb9 docs(plans): tag-browser Task 19 manual smoke PASS 2026-08-01 — online + offline paths verified live 2026-08-01 12:41:24 -04:00
Joseph Doherty 819ca4d7ce docs(plans): retire the folder-hierarchy manual smoke — drag-drop steps obsolete under the [PERM] decision, rest covered by bUnit suites + CLI parity 2026-08-01 12:37:48 -04:00
Joseph Doherty a949fe6f57 docs(plans): retire the env2 + transport manual checklists — core scenario run live 2026-08-01 (caught + fixed the create-missing FK bug), rest covered by automated suites + #31 2026-08-01 12:34:34 -04:00
Joseph Doherty 0c9dffed78 fix(transport): flush created-site ids before the connection pass — create-missing import failed FK 547 on real SQL Server 
Caught live 2026-08-01 importing a bundle into the empty env2 cluster: a
create-missing Site has Id == 0 until SaveChanges on a relational provider,
and ApplyDataConnectionsAsync stamps DataConnection.SiteId as a raw scalar
(no navigation, no EF fix-up), so the insert violated
FK_DataConnections_Sites_SiteId. Every importer integration test runs on
the EF in-memory provider, which assigns ids eagerly on AddAsync — the
exact masking the in-code comment predicted. Fix: one SaveChangesAsync
between the site pass and the connection pass, riding the same outer
transaction (all-or-nothing preserved; the failed live import rolled back
cleanly). Regression test runs the create-missing path on SQLite
(CreateMissingSiteRelationalTests) — red without the fix, green with it.
 2026-08-01 12:32:45 -04:00
Joseph Doherty 1c99d6fa8d docs(grpc-plan): live gate fully closed — ExecuteOpcUa/ExecuteRoute/parked-retry/TriggerSiteFailover all live-proven 2026-08-01 2026-08-01 12:25:27 -04:00
Joseph Doherty 9ab27d5d61 docs(register): rows 27-28 — ES retry config never reaches sites (found live), failover dialog mislabeled Delete 2026-08-01 12:02:23 -04:00
Joseph Doherty f6822f8f45 docs(m10): close follow-up #163 — InstanceConfigureListOverrideTests verified green (#207 re-verified too) 2026-08-01 11:28:46 -04:00
Joseph Doherty 410349767d style(centralui): full-app bg-light/bg-white -> theme-aware utility sweep (M10 residual) 
T34c only fixed the bounded modal-surface offenders. Bootstrap 5.3's bg-light
and bg-white are fixed light values that do NOT flip under [data-bs-theme=dark],
so every remaining use was a dark-mode contrast break. 35 swaps across 19 files:

  surface / <pre> / <code>  bg-light                -> bg-body-secondary
  panel                     bg-white                -> bg-body
  neutral badge             bg-light text-dark      -> bg-secondary-subtle text-secondary-emphasis
  muted badge / input group bg-light text-muted     -> bg-body-secondary text-body-secondary

DELIBERATELY LEFT (7 sites): the neutral member of a status-badge switch or
ternary whose siblings are all solid, non-theme-aware colours (bg-success,
bg-danger, bg-warning) — swapping only the neutral one to a subtle token breaks
the visual weight of the set, so these stay until the whole family is retoned:
  Topology.razor:513 (Current badge) and :588 (InstanceState.NotDeployed)
  InstanceConfigure.razor:1520 (same InstanceState switch)
  NotificationReport.razor StatusBadgeClass fallback
  TransportImport.razor ConflictKind badge fallback
  SecuredWrites.razor text-bg-light fallback (a text-bg-* family with no
    theme-aware member at all)
  Health.razor:377 depth ternary
Also untouched by design: SchemaBuilder.razor:88 (already bg-light-subtle,
theme-aware), bg-dark text-light console panels, and site.css / #reconnect-modal.

InstanceConfigure.razor is edited concurrently elsewhere; its change here is 5
pure class-string swaps on existing lines, no reflow. No test asserted any of
these classes. Also marks all four M10 residuals done in the plan doc.
 2026-08-01 11:28:20 -04:00
Joseph Doherty 316153dc98 feat(centralui): hide OffsetPager on a single page, opt-in — NotificationReport enabled (M10 residual) 
DECISION: a pager whose only two controls are permanently disabled is noise, so
the conventional behaviour ships — the bar is hidden when the result set is a
single page. NotificationReport, which raised the finding, opts in.

RATIONALE for opt-in rather than default-on. OffsetPager is shared, and its
summary span ("Page N of Y · T total") doubles as the ONLY total-count readout
on ConfigurationAuditLog — auto-hiding there would silently delete the result
count for every query returning under a page, and the existing Playwright
fixture asserts "3 total" on exactly such a query. So the behaviour lives in the
shared component (reusable for the next consumer) behind HideWhenSinglePage,
default false, which keeps every current consumer byte-identical.

The guard fires only when single-page-ness is POSITIVELY established:
PageCount <= 1 AND Page <= 1 AND !HasNextPage. A null TotalCount means the host
cannot count, so the bar renders — the controls never vanish while a further
page still exists.

NotificationReport's own Playwright pagination test seeds 51 rows (2 pages) and
is unaffected. 5 new bUnit tests cover hide-on-single-page, hide-on-empty,
show-on-multi-page, show-when-TotalCount-unknown, and the default-off path.
 2026-08-01 11:28:20 -04:00
Joseph Doherty a506b19d17 refactor(centralui): migrate TemplateEdit's four page-embedded modals to the DialogService host (M10 residual) 
TemplateEdit was the last page still hand-rolling its own modal chrome after
M10 — T34c only tokenized its backdrops. All four member-authoring forms
(Attribute, Alarm, Native Alarm Source, Script) now open through
IDialogService.ShowAsync, so the single DialogHost in MainLayout owns the
backdrop, focus trap, Escape and focus restoration.

Pattern copied from the already-migrated pages (MoveDataConnectionDialog +
Templates' Move/Rename dialogs): each body is its own component beside the page
taking a DialogContext<bool>, owning its form state, rendering validation and
server errors INLINE while staying open, and closing with Close(true) only once
the save succeeded — at which point the page reloads. An inline RenderFragment
would NOT have worked: DialogHost renders the captured fragment in its own tree,
so the page's StateHasChanged could never refresh it.

Persistence deliberately stayed on the page (it owns TemplateService, the
inherited-member rules, and the repository-direct native-source path) and is
reached through an OnSaveAsync delegate returning null on success or the message
to display. Behaviour preserved verbatim, including the List-attribute encode +
Decode round-trip check, the name/trigger-type read-only-on-edit rules, the
duplicate-native-source-name guard, and NormalizeExecutionTimeout.

TemplateScriptDialog keeps all four tab panels mounted (Monaco and the JSONJoy
island must not tear down on tab switch) and hosts the Test Run panel, which
needs the live unsaved editor buffer, so it injects ScriptAnalysisService
directly and cancels an in-flight run on dispose.

Extraction moved markup that three structural source-scanning tests pinned;
all three were repointed at the new files rather than weakened:
  - TemplateNativeAlarmSourceEditorTests (+ a new test asserting the form is
    host-mounted and the body renders no chrome of its own)
  - AttributeListEditorTests (list-editor reveal now in the dialog body)
  - TestRunWarningTests (Real I/O warning travelled with the script panel)

Build 0/0; CentralUI.Tests 973/973 green.
 2026-08-01 11:28:20 -04:00
Joseph Doherty fdfd5e1b27 feat(centralui): TreeView full WAI-ARIA keyboard navigation (M10 residual R7) 
TreeView<TItem> handled only Enter/Space on the chevron; the tree itself was
unreachable by keyboard. Implements the WAI-ARIA tree pattern:

- Roving tabindex on li[role=treeitem] — exactly one node is tabbable, so the
  whole tree is a single Tab stop. Target resolved per render: last-focused node
  while still visible -> SelectedKey -> first visible node. Browser focus follows
  via a per-node ElementReference + FocusAsync in OnAfterRenderAsync, guarded by
  the same JSException/JSDisconnectedException/InvalidOperationException triple
  the context-menu focus already used (a no-op under bUnit).
- ArrowDown/ArrowUp move between VISIBLE nodes; ArrowRight expands a collapsed
  branch else moves to first child; ArrowLeft collapses an expanded branch else
  moves to parent; Home/End jump to first/last; Enter/Space activate through the
  SAME path a click takes (OnContentClick / OnCheckboxToggle) so selection
  semantics never diverge between input modes.
- ARIA: aria-level, aria-posinset, aria-setsize added; aria-selected now renders
  true/false on selectable+checkbox trees and stays absent on non-selectable ones.
- Event scoping: @onkeydown:stopPropagation on the li, chevron, checkbox and
  content slot, so nested nodes do not double-handle and consumer controls inside
  NodeContent keep their own key handling. Browser scroll-on-Space/Arrow is
  suppressed by a minimal NATIVE inline onkeydown on the root ul, targeted at
  treeitems only — Blazor's preventDefault directive is all-or-nothing per
  element, so on the li it would trap Tab and cancel Enter/Space on consumer
  buttons. No CSP is configured, so the inline handler runs.

BuildVisibleNodes() mirrors RenderNode's visibility rules and must stay in step.

Tests: 31 new bUnit tests in TreeViewKeyboardNavigationTests; the 47 existing
TreeView tests are unchanged and still green. Docs: new keyboard/a11y section in
docs/components/TreeView.md.
 2026-08-01 11:28:20 -04:00
Joseph Doherty 8aa6bf2270 fix(audit): populate ParentExecutionId on alarm-triggered script runs 
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.

Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:

  ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
    → SetStaticAttributeCommand.SourceExecutionId
    → AttributeValueChanged.SourceExecutionId   (InstanceActor static-write path)
    → AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext

All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.

Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).

Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.

Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
 2026-08-01 11:22:07 -04:00
Joseph Doherty 88638d774a feat(cli,management): close area-move and template-folder CLI parity gaps 
Two verified-absent parity gaps between the service layer and the CLI /
ManagementActor command surface, both left as follow-ups by the 2026-05-11
design plans.

(1) area move. AreaService.MoveAreaAsync had existed since the deployment
topology page shipped but was reachable only from the Blazor UI. Adds
MoveAreaCommand(AreaId, NewParentAreaId?) to Commons, a ManagementActor
dispatch arm delegating straight to AreaService.MoveAreaAsync (not-found /
self-parent / descendant-cycle / cross-site / name-collision all surface as
the standard curated ManagementCommandException failure response; the service
writes its own "Move" audit row), and the CLI verb `site area move --id
[--parent-id]`. Omitting --parent-id moves the area to the site root, matching
the command's nullable NewParentAreaId. The command carries the SAME any-of
[Designer, Deployer] gate as CreateArea/UpdateArea/DeleteArea (arch-review C6):
re-parenting is the same structural authoring act, exposed on the same two
surfaces. Placed under the existing `site area` group rather than a new
top-level `area` group, alongside its create/update/delete siblings.

(2) template folder verbs. The five folder management commands have been
handled by ManagementActor since the folder-hierarchy plan, but the promised
CLI surface was never written. Adds `template folder
list|create|rename|move|reorder|delete` mapping 1:1 onto ListTemplateFolders /
CreateTemplateFolder / RenameTemplateFolder / MoveTemplateFolder /
ReorderTemplateFolder / DeleteTemplateFolder. --parent-id is omitted to target
the tree root; --direction takes the lowercase literals up/down, validated at
parse time by AcceptOnlyFromAmong (same case-sensitive contract as the audit
--channel/--kind/--status options).

Follow-on updates: the frozen authorization matrix gains its MoveArea entry
(reflection-driven, so a missing entry would have failed CI); CommandTreeTests
pins both new verb sets plus the omit-parent-id-means-root parse behaviour and
registry round-trips; ManagementActorTests covers the MoveArea role gate and
the delegate-to-service success/root/cycle/not-found paths; the CLI README and
Component-ManagementService.md document the new surface (the latter also gained
the previously-undocumented ReorderTemplateFolder); both plan docs' follow-up
lines are marked done.
 2026-08-01 11:15:24 -04:00
Joseph Doherty e0851e3e17 docs(register): close row 12 — InstanceConfigure native-alarm-source CSV upload shipped 2026-08-01 11:12:10 -04:00
Joseph Doherty a26d6ba317 feat(central-ui): native-alarm-source CSV import on InstanceConfigure 
The Native Alarm Source Overrides card had no bulk affordance — the CSV path
shipped CLI-only (instance native-alarm-source import --file), so the UI could
only retarget one source at a time inline.

Adds a second InputFile on that card, mirroring the attribute importer's UX
(hidden input behind a button-styled label, 512 KB cap, success/error alert
with the per-line error list, toast). Parsing reuses the SHARED
NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the
new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same
batch rules the server enforces in
ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must
resolve, must not be template-locked, and may appear at most once; any error
rejects the whole file and applies nothing.

Semantics match the CLI: merge, not full replace — sources absent from the
file keep their existing override, a blank field keeps the inherited value,
and an all-blank row clears that source's override (equivalent to the CLI's
all-null override row, without leaving a dead row behind).

Persistence reuses the inline editor's path: SaveNativeOverride's upsert body
is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the
import commits the validated batch in one SaveChangesAsync — no new server
method, no duplicated parsing.

Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field
inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source
and parser-error rejection, plus structural pins on the InputFile wiring.
Docs: Component-CentralUI.md native-alarm-source card gains the import bullet.
 2026-08-01 11:11:40 -04:00
Joseph Doherty 697be0ce43 docs(topology): fix stale keep-oldest reference in Site Cluster Behavior — clusters run auto-down since 2026-07-21 2026-08-01 11:10:48 -04:00
Joseph Doherty 4558bc3b1f docs(register): track 4 untracked deferrals, split the failover/perf row, close #18 
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).

Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.

Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
 2026-08-01 11:10:19 -04:00
Joseph Doherty 2d03f2d507 fix(site-runtime): reconcile artifact deletions on apply — central deletes no longer orphan site rows 
The artifact apply (DeploymentManagerActor.HandleDeployArtifacts) was
upsert-only, so deleting an external system (or shared script, DB connection,
data connection) centrally never removed the site's SQLite row — a deleted
external system stayed callable from site scripts forever. Central always
ships the COMPLETE set of each artifact class (ArtifactDeploymentService
GetAll* snapshots; the wire's presence-tracking wrapper lists preserve
null-vs-empty), so the site now applies upsert-then-reconcile: after storing
the incoming set, SiteStorageService.DeleteRowsExceptAsync removes any stored
row absent from it, per artifact table. A null list still means 'field not
shipped' and touches nothing.

Runtime cleanup rides along: a reconciled-away shared script is unregistered
from the compiled SharedScriptLibrary (a stale delegate would stay callable
until restart), and a removed data connection is evicted from the DCL hash
cache and its live connection actor stopped via the previously-caller-less
RemoveConnectionCommand — both on the actor thread via the extended
ApplyArtifactDataConnectionsToDcl message. All four tables are
RegisterReplicated, so the deletes reach the standby as ordinary CDC row
tombstones.

Tests: storage-level reconcile per table (incl. empty-set-deletes-all and
idempotency) in ArtifactStorageTests; actor-level pins in
DeploymentManagerActorTests (orphan delete, null-set no-op, library
unregistration, DCL stop for the removed connection only). Docs:
Component-DeploymentManager + Component-SiteRuntime record the
full-set/reconcile semantics.
 2026-08-01 10:54:18 -04:00
Joseph Doherty 0123b68719 docs(plans): MES alarm-status API — all 7 open design questions decided 
Design review 2026-08-01: (1) MES relevance = dedicated severity band 900-999
(script constants, real IsFlaggedForMES predicate); (2) MESReceiver version
DROPPED — CvdReactor is the only implementation, router returns a clean
not-supported error elsewhere; (3) Description = Message else AlarmTypeName;
(4) enrich the native mirror NOW with an additive AckTime (proto +
native_alarm_state + DCL stamping); (5) script names match the endpoints;
(6) shared ReactorAlarms on both sides, suffix behavior as planned;
(7) existing MES API key, no new roles. Plan is now ready to execute.
 2026-08-01 10:29:44 -04:00
Joseph Doherty d1ade5653b docs(plans): bookkeeping sync — reconcile stale trackers with merged code 
A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:

- Flip ~380 stale task statuses to completed across March/May/June/July
  trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
  LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
  code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
  HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
  template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
  (premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
  -> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
  scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
  merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
  sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
  TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
  #28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
  seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
  requirements-traceability 'Pending' clarified as frozen plan-generation
  status.

Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.
 2026-08-01 08:53:56 -04:00
Joseph Doherty 663d03e89c Merge branch 'fix/cached-telemetry-drain-hot-loop' 2026-07-27 15:50:42 -04:00
Joseph Doherty 63c16d6912 refactor(comm): retire ClusterClient naming after the gRPC cutover 
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.

Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.

Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
 2026-07-27 15:40:01 -04:00
Joseph Doherty b3dc17a5fe Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0) 2026-07-24 13:35:27 -04:00
Joseph Doherty 1c9159a7ce fix(tests): qualify Health page type — Communication.Health namespace shadowed it 2026-07-24 13:35:24 -04:00
Joseph Doherty 5d4853a1f0 refactor(health): adopt the shared active-node check (Health 0.3.0) 
OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck
selected by cluster leadership (review 01 [High]): leadership is address-ordered
and diverges from singleton placement after a restart, and during a partition
both sides compute themselves leader so Traefik served both. Health 0.3.0 makes
the shared check age-based — it is now this host's own rule, promoted — so the
private copy is deleted and central registers the shared type.

ActiveNodeEvaluator, the "THE single definition of active node" this repo already
maintained, now delegates to the shared ClusterActiveNode rather than
re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp,
the inbound-API gate and the /health/active tier on one rule, and it means the
rule is shared with OtOpcUa, which had independently written a third copy.
Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering
trade-off is recorded at the PackageReference.

SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the
rule — it is a thin adapter over the site's own IClusterNodeProvider, which
scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering
the shared check directly would have to re-derive the site role and would lose the
property that the tier and singleton placement come from the same provider.
Central stays unscoped: every central member competes for one active slot.

No behaviour change on any node — same rule, one implementation instead of two.

Verified: Host.Tests 439/439, Communication ActiveNode 2/2.
 2026-07-24 13:21:58 -04:00
Joseph Doherty c1216044a4 docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings 
Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the
bootstrap guard' subsection (dark switch, founder/probe semantics, config table,
accepted trade) plus a forward reference from Dual-Node Recovery. Host
appsettings.Central/Site.json carry a disabled BootstrapGuard block with a
_bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover
note added.
 2026-07-24 08:57:44 -04:00
Joseph Doherty 248676ed16 feat(cluster): simultaneous-cold-start split-brain guard (#33) 
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
 2026-07-24 08:55:48 -04:00
Joseph Doherty 47850c0f53 feat(health): serve MapZbHealth on site nodes 
Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and
/metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site
node whether it was ready or which half of the pair was active. The family
overview dashboard probes every instance the same way, and this is the one gap.

Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so
the composition-root tests that build this graph actually cover them) and mapped
by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside
/metrics:

  akka-cluster  [Ready]  the shared AkkaClusterHealthCheck. Also carries the
                         cluster-view data (leader/memberCount/...) the dashboard
                         reads, free with ZB.MOM.WW.Health 0.2.0.
  localdb       [Ready]  NEW SiteLocalDbHealthCheck — SELECT 1 through the
                         registered ILocalDb. Site has no EF context; central's
                         DatabaseHealthCheck<ScadaBridgeDbContext> is central-only.
                         Replication state rides along as data ENRICHMENT only:
                         replication is default-OFF, so failing on it would mark
                         every correctly-configured node unready.
  active-node   [Active] NEW SitePairActiveNodeHealthCheck, delegating to
                         IClusterNodeProvider.SelfIsPrimary. Deliberately NOT
                         central's OldestNodeActiveHealthCheck: that one calls
                         SelfIsOldest(cluster) with no role argument and would
                         compute "oldest" across the wrong member set on a mesh
                         carrying more than one site.

Anonymous, as central's are: the site pipeline runs no authentication middleware
and has no FallbackPolicy, so nothing extra was needed.

Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free).

Tests: SiteHealthCheckTests builds the REAL site container and activates every
registration through its factory (exact-set names, one tier tag each, resolved
types, central-only checks absent behind a positive control) plus behaviour for
both new checks. SiteHealthEndpointTests boots the real site Program over
WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting
MapZbHealth would leave every registration test green while site nodes 404'd.

Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.
 2026-07-24 05:54:30 -04:00
Joseph Doherty bb138d5254 docs: add env_variables.md — full environment-variable inventory 
Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI,
DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra
containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a
shipped appsettings key already exists for it, consumer, purpose, potential
values, and required-ness. Compiled from GetEnvironmentVariable call sites, the
Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1,
and all appsettings*.json.
 2026-07-24 05:02:24 -04:00
Joseph Doherty 9a12826172 docs(plans): OtOpcUa v3 native-alarm B/C live-gate PASS (#14) — no code needed; item C premise obsolete (Gitea #17) 2026-07-24 04:33:20 -04:00
Joseph Doherty 86d129de6c fix(dcl): harden endpoint host-rewrite — IPv6 brackets + portless URLs 
Code review of a1abbff7 found two edge cases: a discovery/advertised URL with no
explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an
IPv6 literal host could lose/double its brackets. Omit the port when neither URL
carries one; bracket an IPv6 host only when missing. +4 tests (12 total).
 2026-07-23 16:58:44 -04:00
Joseph Doherty dbec3ee263 docs(plans): OtOpcUa v3 raw-path live-gate PASS (#14) — fix validated live, B/C deferred 2026-07-23 16:54:48 -04:00
Joseph Doherty a1abbff760 fix(dcl): rewrite advertised OPC UA endpoint host to the reachable one 
An OPC UA server advertises its base address from its own config, not the route
the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT
hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl
verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and
the connect failed. RealOpcUaClient now swaps the advertised authority for the
reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path;
no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).
 2026-07-23 16:43:45 -04:00
Joseph Doherty 266f001a2e docs(plans): OtOpcUa v3 dual-namespace cutover scope + phase-2 plan (#14) 2026-07-23 15:22:22 -04:00
Joseph Doherty e04c2617cc feat(ui): warn on bare ns= node bindings in the browse picker (v3 cutover #14) 2026-07-23 15:18:36 -04:00
Joseph Doherty 97afa84fcd docs(dcl): sweep ns= examples to durable nsu= form (v3 cutover #14) 2026-07-23 15:17:13 -04:00
Joseph Doherty 0eb44314cb feat(dcl): add OpcUaReferenceForm.IsDurable — flags bare ns= bindings (v3 cutover #14) 2026-07-23 15:16:47 -04:00
Joseph Doherty a5256e9b12 chore(comm): delete dead IntegrationCallRequest routing (Gitea #32) 
'Pattern 4: Integration Routing' — RouteIntegrationCallAsync →
SiteEnvelope(IntegrationCallRequest) → SiteCommunicationActor integration
handler — was plumbed end to end but connected at neither end: no
producer (zero callers) and no handler (AkkaHostedService never
registered LocalHandlerType.Integration). It was an early scaffold the
architecture routed around — the brokered External→Central→Site→Central
round-trip is served by the Inbound API's routed-site-script path (the
RouteTo* verbs, driven from CommunicationServiceInstanceRouter), which is
live, tested, and shares IntegrationTimeout.

Decision (#32): delete. Removed the IntegrationCall{Request,Response}
messages, RouteIntegrationCallAsync, the SiteCommunicationActor receive
block + _integrationHandler field + LocalHandlerType.Integration, and the
four tests that covered them (2 actor, 2 message-contract, 1 dispatcher-
reject, 1 mapper-reject). KEPT IntegrationTimeout — it is the live
timeout for the RouteTo* verbs. Updated the exclusion-narrative comments
(proto/mapper/dispatcher), design §4, the components doc timeout table,
and marked the known-issue RESOLVED.

Full solution build clean (0/0); Communication 634 + Host 421 green.
Net -172/+20 across 14 files. Not in the gRPC proto (was deliberately
excluded there), so no wire-format change.
 2026-07-23 14:27:32 -04:00
Joseph Doherty e0f105c3b3 docs(env2): live-gate docker-env2 on the gRPC PSK build — PASS 3/3 (Gitea #31) 
Rebuilt scadabridge:latest from main @ 8524a7f7 and recreated only the
env2 containers. All three gate checks pass on site-x:
1. both site nodes boot with the key (StartupValidator fail-closed →
   reaching 'Application started' proves the key present);
2. control-plane PSK auth: no-header / wrong-key ⇒ PermissionDenied,
   correct key ⇒ success, on both nodes (:9123, :9124);
3. LocalDb unaffected (local-only; 0 errors, healthy boot).
Bonus: central registers site-x online via gRPC heartbeat; no real
ClusterClient/receptionist (only the benign ClusterClientSiteAuditClient
label, same as the primary rig). Noted a seed-data gap (ScadaBridgeConfig2
dbo.Sites is empty) — orthogonal to the transport.
 2026-07-23 14:10:22 -04:00
Joseph Doherty 8524a7f746 test(sms-e2e): valid Twilio SID fixture so the create path passes (Gitea #29) 
TestAccountSid was 'ACtest123' (AC + 6 chars). The management create
path validates the Account SID against ^AC[0-9a-fA-F]{32}$
(ManagementActor.cs:2205, added 2026-07-10, commit 40088a21) while the
fixture predates the guard (2026-06-19) — so the create was rejected and
the config-card + secret-non-leak assertions have been red/inert since.

Use AC + 32 hex (AC00000000000000000000000000000001) so the create
succeeds and the downstream Auth-Token-non-leak assertion actually runs
again. Unit-level RepositoryCoverageTests keep their short SIDs — they
construct SmsConfiguration directly and never hit the validated path.
 2026-07-23 14:03:49 -04:00
Joseph Doherty c4dcd9bc02 fix(transport): run bundle import inside the execution strategy (Gitea #28) 
BundleImporter.ApplyAsync opened a user-initiated transaction directly,
but the central ConfigurationDb context is configured with
EnableRetryOnFailure. SqlServerRetryingExecutionStrategy rejects
user-initiated transactions, so every bundle import threw on any real
SQL Server ('...does not support user-initiated transactions.'). The
whole Transport suite ran on the in-memory provider (no retrying
strategy, BeginTransaction is a no-op) so it never surfaced.

Extract the transactional apply into ApplyMergeAsync and drive it via
_dbContext.Database.CreateExecutionStrategy().ExecuteAsync, so the
strategy owns the BeginTransaction -> apply -> Commit unit as one
retriable block. The delegate resets per-attempt state (change tracker
+ ApplyMergeAsync rebuilds its own summary/resolution accumulators) so a
retried attempt cannot double-apply; a rollback failure is captured and
surfaced on the BundleImportFailed audit row exactly as before. Post-
commit side effects (ScriptArtifactsChanged publish, session zero/
remove) moved outside the retriable delegate so they run once.

Regression test: BundleImporterRetryingStrategyTests runs the import on
SQLite with a retrying execution strategy (RetriesOnFailure == true,
arming the same guard as production). It fails with the exact production
error on the pre-fix code and passes after. Existing rollback/apply
contracts unchanged (104 integration + 154 unit tests green).
 2026-07-23 13:55:09 -04:00
Joseph Doherty 9f91e84d83 docs(grpc): Phase 5 live gate — PASS (8/8), migration complete 
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.

Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
 2026-07-23 13:35:16 -04:00
Joseph Doherty 7fd5cb2b56 feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only 
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.

Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
  ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
  AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
  coexistence flags; the CentralTransportMode/SiteTransportKind enums

gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.

Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).

Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.

Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.

Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
 2026-07-23 12:54:32 -04:00
Joseph Doherty 3a8ddb7087 docs(known-issues): link IntegrationCallRequest dead-code note to Gitea #32 2026-07-23 11:52:52 -04:00
Joseph Doherty 54da10dc00 docs(grpc): Phase 3 live gate — PASS; central→site command cutover 
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.

Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
  (SoakNotify instances survived the recreate as Enabled)

Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
  TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
  retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
  (SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
  (notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes

ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
 2026-07-23 11:45:12 -04:00
Joseph Doherty a54602c14a docs(grpc): Phase 2 live gate — PASS; site→central S&F cutover soak 
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.

Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
  logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
  bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
  (oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
  active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
  whole dead window, 216==216 distinct — zero loss, zero dupes

Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
 2026-07-23 11:31:09 -04:00
Joseph Doherty 2fa5e93c73 docs(plans): tick 1B DoD (proportionate rig gate PASS) 2026-07-22 22:38:38 -04:00
Joseph Doherty 01693b13db docs(grpc): Phase 1B live gate — PASS (proportionate); central→site rides authenticated gRPC for 3 sites; records the central-wide SiteTransport finding 2026-07-22 22:38:21 -04:00
Joseph Doherty 86ad4d5c8e feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka) 
Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.

- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
  (the _siteClients lookup + ClusterClient.Send with the reply-to sender
  preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
  SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
  ControlPlaneCredentials, per-command deadlines set EQUAL to today's
  CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
  query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
  QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
  WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
  (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
  may have run), background failback probe to the preferred node with 1s→60s
  doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
  loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.

Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.
 2026-07-22 20:09:43 -04:00
Joseph Doherty 518c699b90 feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2) 
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.

Decisions worth recording:

- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
  Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
  handlers with the exact same "handler not available" replies; the parked
  handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
  proxy — pinned by a dispatcher test that asserts the target is the parked probe
  and NOT the dm proxy.

- Sender preservation intact. The actor's command handlers became thin
  DispatchCommand delegations that still Forward (central Ask → reply routes
  straight back); the existing SiteCommunicationActorTests pass unchanged, which
  is the regression guard for that plumbing. UnsubscribeDebugView keeps its
  fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
  synthetic UnsubscribeDebugViewAck so a unary RPC still answers.

- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
  standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
  CommitLeave; the gRPC service returns the ack, then schedules the real
  Cluster.Leave — so a caller reaching the very node about to leave still gets its
  ack instead of a broken stream. The actor path keeps today's coupled
  resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
  immaterial). Proven at both levels: a dispatcher test asserts the ack is built
  before CommitLeave runs, and a TestServer test asserts the recorded seam order
  is resolve-then-leave.

- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
  DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
  one-public-ctor invariant and its test stay green.

Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
 2026-07-22 20:07:58 -04:00
Joseph Doherty 59b13d317b feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens 
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.

Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).

Contract decisions worth knowing:

- Nullable COLLECTIONS ride in per-collection wrapper messages
  (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
  RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
  and that distinction is live at the site — the same silent-data-loss class the
  transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
  and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
  AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
  TargetValueEncoded, where "wait for the empty string" is a real target, so it
  carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
  no stock wrapper). Enum translation is an explicit switch in both directions —
  never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
  enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
  rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
  return values, attribute values, tag read/write values) as a type-tagged union
  so a boxed value keeps its runtime CLR type, as it does today under Akka's
  type-preserving JSON serializer. Dates ride as invariant round-trip strings,
  not Timestamp, which would silently normalise away DateTime.Kind and
  DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
  falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
  than reusing sitestream's AlarmStateUpdate, which flattens values to display
  strings — right for a live stream, lossy for a snapshot the UI treats as
  authoritative. The encoder omits an AlarmStateChanged.Condition that already
  equals the record's derived default, so computed alarms round-trip exactly
  (record equality compares the nullable backing field, not the property).

Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).

Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
 2026-07-22 20:04:23 -04:00
Joseph Doherty aa60f43866 Merge Phase 1A: site→central control plane over gRPC (PR #26) 2026-07-22 20:01:42 -04:00
Joseph Doherty f7c7811940 docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect 2026-07-22 19:55:09 -04:00
Joseph Doherty 0e162cb250 fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression) 
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
 2026-07-22 19:50:32 -04:00
Joseph Doherty aa49a1d078 docs(plans): tick T1B.3/T1B.4 — central site-command transport seam complete (3be85f19) 2026-07-22 19:43:58 -04:00
Joseph Doherty 81ced76654 docs(plans): tick T1A.3/T1A.4 — site ICentralTransport seam complete (33b15f10) 2026-07-22 19:31:05 -04:00
Joseph Doherty 33b15f10a4 feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3) 
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
 2026-07-22 19:29:14 -04:00
Joseph Doherty 9f2c96f486 docs(plans): tick T1B.2 — SiteCommandDispatcher + site gRPC command service (cd6c20e1) 2026-07-22 19:12:45 -04:00
Joseph Doherty fc398b47d3 docs(plans): tick T1A.2 — central CentralControlService hosting + per-site auth (780bb9c3) 2026-07-22 18:55:28 -04:00
Joseph Doherty 780bb9c369 feat(grpc): host CentralControlService on the central node (T1A.2) 
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
 2026-07-22 18:53:59 -04:00
Joseph Doherty c90b353820 docs(plans): tick T1B.1 — site_command.proto + mapper + 194 goldens (a7481174) 2026-07-22 18:41:22 -04:00
Joseph Doherty c615ba5f78 docs(plans): tick T1A.1 — central_control.proto + mapper + 32 goldens (d7455577) 2026-07-22 18:31:18 -04:00
Joseph Doherty d7455577a8 feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1) 
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
 2026-07-22 18:28:27 -04:00
Joseph Doherty aa7c5cd138 docs(plans): tick Phase 0 DoD — PR #25 merged to main @ 3fa95555 2026-07-22 18:12:01 -04:00
Joseph Doherty 3fa955556d docs(grpc): record the Playwright result and root-cause both failures 
Phase 0's gate doc now carries the full suite picture, not just the rig checks.

Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):

- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
  user-initiated transaction while the central context has EnableRetryOnFailure,
  so SqlServerRetryingExecutionStrategy refuses the split query inside it and
  bundle import fails against real MS SQL. The unit/integration suite cannot see
  it -- the in-memory EF provider has no retrying strategy and BeginTransaction
  is a no-op there.

- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
  ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
  has also silenced everything after the toast assertion -- including the
  secret-non-leak check on the Auth Token.

Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.

Neither is fixed here; both are out of scope for a PSK-auth branch.
 2026-07-22 18:09:07 -04:00
Joseph Doherty 6ef8c7d70a docs(grpc): Phase 0 live gate PASS — record results, the inert-gate defect, and the trap for phases 1A/1B 
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.

Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
 2026-07-22 18:01:11 -04:00
Joseph Doherty 228ff8b428 fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert 
Caught by the Phase 0 live gate, not by the suite.

Grpc.AspNetCore registers the interceptor BY TYPE, and
InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all
given argument types have been found" when more than one public constructor is
applicable. The interceptor had two: the DI one and a prefix-set overload added
for later phases.

The failure mode is nasty. The throw happens inside the interceptor pipeline on
every call, so nothing fails at startup — the site node boots, joins, reports
healthy. Every gated call then dies with Unknown / "Exception was thrown by
handler", which reads as a handler bug rather than an auth bug. And it fails
OPEN in the sense that matters least and closed in the sense that matters most:
no call is ever authorized, but no call is ever correctly REFUSED either, so the
rig showed identical errors for a correct key, a wrong key and no key at all.
Live evidence, site-a: three PullAuditEvents calls, three identical
InvalidOperationExceptions in the node log.

Fix: the prefix-set constructor is internal (Host.Tests already has
InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding
a second public registration shape.

Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests
registered the interceptor with AddSingleton alongside AddGrpc, so DI handed
back the instance and Grpc.AspNetCore's activation path — the thing that throws
— never ran. The harness now registers exactly as Program.cs does, by type and
not in DI. Plus a direct reflection assertion that the type has exactly one
public constructor, since that is the real invariant and it is cheap to pin.
 2026-07-22 17:56:51 -04:00
Joseph Doherty 2ee84af1c0 feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration 
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
 2026-07-22 17:51:09 -04:00
Joseph Doherty f1ad967083 docs(plans): ClusterClient→gRPC-only migration plan (phases 0–5 + tasklist) 
Complete executable plan: PSK-from-Secrets auth (Phase 0), central_control +
site_command proto contracts behind transport seams inside the two
communication actors (1A ∥ 1B in worktrees), per-direction cutover flags,
ClusterClient/receptionist deletion, 8-check live gate. Design doc:
scadaproj/scadabridge_clusterclient_to_grpc.md (§7 = deep-dive corrections).
 2026-07-22 17:10:33 -04:00
Joseph Doherty 654df8abc2 docs(cluster): site-pair manual failover runbook + component spec 2026-07-22 07:49:51 -04:00
Joseph Doherty c8e2f4da02 feat(cluster): site-pair manual failover relayed from the central UI (Task 10) 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
 2026-07-22 07:48:56 -04:00
Joseph Doherty d66e0d585f chore(plans): mark self-first ordering + manual failover tasks complete (live gate PASS) 2026-07-22 07:14:11 -04:00
Joseph Doherty caf14a3e03 feat(ui): admin manual-failover control on the health page 
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.

- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
  intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
  the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
  hand over, in-flight work on the active node is interrupted, and THIS PAGE
  will disconnect and reconnect against the new active node -- Traefik routes
  the UI to the node being restarted, so a working failover otherwise reads as
  a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
  reports a failover that did not happen.

7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.

Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
 2026-07-22 07:00:10 -04:00
Joseph Doherty f679d5c749 feat(cluster): manual central failover service — graceful Leave of the oldest Up member 
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
 2026-07-22 06:55:15 -04:00
Joseph Doherty eca69505bc docs(plans): record the self-form watchdog rejection and the self-first ordering that replaced it 2026-07-22 06:33:28 -04:00
Joseph Doherty 4a6341d871 feat(cluster): self-first seed ordering closes the boot-alone outage gap 
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
 2026-07-22 06:32:00 -04:00
Joseph Doherty 69b3ccfc37 docs(components): correct the three component docs against the code they describe 
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.

ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.

Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.

DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.

Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.

Docs only; no code changed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 18:45:56 -04:00
Joseph Doherty cf3bd52f93 feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety) 
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
 2026-07-21 10:53:40 -04:00
Joseph Doherty dced0d2794 fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — four new NU1903 advisories on the 10.0.7 DataProtection transitive broke fresh restores (docker image build) 
Same pattern as the SQLitePCLRaw pin: direct PackageReference at the chain's entry
project (ConfigurationDatabase). Bumping the DataProtection parent instead was tried
and rejected — 10.0.10 floors Microsoft.Extensions.*/EF at 10.0.10 (NU1605 cascade).
 2026-07-21 10:53:26 -04:00
Joseph Doherty 34227991ea docs(claude): correct five understatements about the inter-cluster boundary 
Found while OtOpcUa researched this repo as the model for its own per-cluster
mesh work. Each was verified against the code, not inferred from the doc.

Three transports cross the boundary, not two: ClusterClient, gRPC, and plain
token-gated HTTP for the deploy config fetch (DeploymentConfigEndpoints,
X-Deployment-Token, AllowAnonymous with the per-deployment token as the entire
security boundary).

The gRPC direction is inverted from the data flow. Data moves site to central,
but each SITE hosts the gRPC server and central dials in — MapGrpcService appears
only in the Site branch of Program.cs. There is no gRPC server on central, which
is why the two Ingest* RPCs are dead in practice. Also records the 6-RPC surface
(the doc implied 2), the (site, endpoint) factory key that fixed an arch-review
High, and the vendored-generated-code caveat.

Active/standby is ActiveNodeEvaluator.SelfIsOldestUp — the OLDEST Up member, and
explicitly never cluster.State.Leader, because leadership diverges from singleton
placement permanently after a restart-and-rejoin and both sides claim it during a
partition. The equivalence oldest-Up == singleton placement is the design, and it
was not stated anywhere in this file.

All clusters share one ActorSystem name ("scadabridge", hardcoded at
AkkaHostedService.cs:191); they are separate clusters only by seed partitioning.
Required, not incidental — Akka.Remote address matching means ClusterClient could
not reach a differently-named system. Site nodes also carry two roles, base plus
site-{SiteId}, with singletons scoped to the site-specific one.

Neither inter-cluster transport is authenticated or encrypted: no Akka TLS or
secure cookie, gRPC is h2c, and LocalDbSyncAuthInterceptor gates only the LocalDb
sync path — so the whole SiteStreamService surface, including the two Pull RPCs
returning audit rows, is reachable by anyone who can hit :8083. The boundary
assumes a trusted network; that assumption deserves to be explicit.

Also promotes two things from docs/ into CLAUDE.md because they change decisions:
the default 128 KB Akka frame size with log-frame-size-exceeding off and no custom
serializer (a silent single-message drop that leaves the association healthy), and
the registered two-node keep-oldest total-outage gap, whose own drill has a mode
existing "to make the registered gap observable — not to pretend it is covered."

Committed to a branch rather than main; merge at your discretion.
 2026-07-21 10:00:01 -04:00
Joseph Doherty 9d925c3347 Merge chore/localdb-0.1.3: pin ZB.MOM.WW.LocalDb 0.1.3 
Picks up two rebuilt-peer replication fixes found by the OtOpcUa LocalDb
Phase 1 live gate. They matter more here than in OtOpcUa, which replicates 2
tables to this repo's 10:

- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
  detection read that as 'no gap possible', so a node whose database was lost
  rejoined empty and stayed empty until the next deploy.
- 0.1.3 — with back-fill working, the rebuilt node's own writes were silently
  dropped until its restarted seq counter climbed past the peer's stale
  watermark.

Build clean; SiteRuntime 512 green.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:19:13 -04:00
Joseph Doherty 8c6fc2f886 chore(localdb): pin ZB.MOM.WW.LocalDb 0.1.3 (rebuilt-peer replication fixes) 
Two fixes, both about a node whose LocalDb file is lost. They matter more here
than in OtOpcUa, which replicates 2 tables to this repo's 10.

0.1.2 — a converged pair prunes every oplog row on ack, and snapshot detection
read an empty oplog as "no gap possible". The steady state of a healthy pair
was the one state from which a rebuilt node could never be healed: it rejoined
empty and stayed empty until the next deploy.

0.1.3 — with back-fill working, the rebuilt node's OWN writes turned out to be
silently dropped: last_applied_remote_seq is a watermark in the peer's seq
space, and a rebuilt peer numbers from 1, so the healthy node's stale watermark
made the sender skip its whole oplog.

Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig; the
second only became reachable once the first was fixed.

Build clean; SiteRuntime 512, SiteEventLogging 70 green (Host 330,
HealthMonitoring 97 green on 0.1.2, unchanged by the pin).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:18:02 -04:00
475 changed files with 77664 additions and 9850 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -55,3 +55,4 @@ docker-env2/*/data/
# Sister-project deployment artifacts (not part of this solution) # Sister-project deployment artifacts (not part of this solution)
/deploy/ /deploy/
email_details.txt
CLAUDE.md
+32 -71
View File
@@ -8,29 +8,12 @@ When a change is requested, the default assumption is: update the design doc *an
### Top-level directories ### Top-level directories
- `src/` — C#/.NET implementation, one project per component (e.g. `ZB.MOM.WW.ScadaBridge.AuditLog`, `ZB.MOM.WW.ScadaBridge.NotificationOutbox`, `ZB.MOM.WW.ScadaBridge.SiteCallAudit`, `ZB.MOM.WW.ScadaBridge.CentralUI`, `ZB.MOM.WW.ScadaBridge.Host`, …). Solution file: `ZB.MOM.WW.ScadaBridge.slnx`. Layout is self-describing (`ls`, plus each directory's own README). Only the non-obvious parts are recorded here:
- `tests/` — Test projects (unit + integration).
- `docs/` — Design documentation: `docs/requirements/` (high-level + per-component specs), `docs/test_infra/` (test infrastructure), `docs/plans/` (design-decision and implementation-plan docs). The spec the code implements.
- `docker/` — 8-node cluster topology (2 central + 3 sites), `deploy.sh`, per-node `appsettings.*.json`. See [`docker/README.md`](docker/README.md) for setup, ports, and management commands. Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — Minimal second cluster topology (2 central + 1 site × 2 nodes), runs concurrently with `docker/` on host ports 91XX. Built specifically for testing the Transport (#24) feature with two real environments. See [`docker-env2/README.md`](docker-env2/README.md). Rebuild + redeploy with `bash docker-env2/deploy.sh`.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`).
- `deploy/` — Production/on-host deployment artifacts (e.g. `deploy/wonder-app-vd03/`: `appsettings.Central.json`, `appsettings.Site.json`, `install.ps1`/`uninstall.ps1`, `RUNBOOK.md`).
- `deployments/` — Deployment topology notes (`docker-cluster.md`, `docker-cluster-env2.md`, `README.md`).
- `code-reviews/` — Per-component code-review notes (one folder per component, plus `_template`).
- `tools/` — Repo maintenance/utility scripts (e.g. `rename-to-scadabridge.sh`).
- `AkkaDotNet/` — Akka.NET reference documentation and best-practices notes.
- `deprecated/` — Retired docs/notes kept for reference.
- `logs/` — Local runtime log output.
- `vendor/` — Vendored third-party assets (currently an empty placeholder).
- `.claude/` — Claude Code project config (settings, skills, agents).
### Key documents - `docker/` — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — minimal second cluster (2 central + 1 site × 2 nodes) that runs **concurrently** with `docker/` on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with `bash docker-env2/deploy.sh`.
- `README.md` — Master index with component table and architecture diagrams. - `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`), and the same applies to `docs/test_infra/test_infra.md`, which still reads as if it were a local container.
- `docs/requirements/HighLevelReqs.md` — Complete high-level requirements covering all functional areas. - `deprecated/` — retired docs/notes kept only for reference; not current.
- `docs/requirements/Component-*.md` — Individual component design documents (one per component) — the spec the code implements.
- `docs/test_infra/test_infra.md` — Master test infrastructure doc (OPC UA, MS SQL, SMTP, REST API, Traefik). LDAP is the shared GLAuth on `10.100.0.35:3893` (not a local infra container; see `scadaproj/infra/glauth/`).
- `docs/plans/` — Design decision and implementation-plan documents from refinement sessions.
## Sister Projects ## Sister Projects
@@ -70,35 +53,11 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Run tests with `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. `ZB.MOM.WW.ScadaBridge.CLI.Tests` is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too. - Run tests with `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. `ZB.MOM.WW.ScadaBridge.CLI.Tests` is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
- **Propagate cross-repo changes to the umbrella index.** This repo is indexed by the parent workspace `~/Desktop/scadaproj`. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the **ScadaBridge entry in [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. `MxAccessGateway`/`HistorianGateway`.) - **Propagate cross-repo changes to the umbrella index.** This repo is indexed by the parent workspace `~/Desktop/scadaproj`. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the **ScadaBridge entry in [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. `MxAccessGateway`/`HistorianGateway`.)
## Current Component List (27 components) ## Component List
1. Template Engine — Template modeling, inheritance, composition, validation, flattening, diffs. 27 components. The catalog — what each one is and its key design commitments — lives in the
2. Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle. `scadabridge-components` skill (`.claude/skills/scadabridge-components/SKILL.md`); the authoritative
3. Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream. spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carries the component table.
4. Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
5. Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
6. Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
7. External System Gateway — External system definitions, API method invocation, database connections.
8. Notification Service — Central-only notification-list and SMTP definitions, per-type delivery adapters (sites no longer deliver notifications).
9. Central UI — Web-based management interface, all workflows.
10. Security & Auth — LDAP/AD authentication, role-based authorization, site-scoped permissions.
11. Health Monitoring — Site health metrics collection and central reporting.
12. Site Event Logging — Local operational event logs at sites with central query access.
13. Cluster Infrastructure — Akka.NET cluster setup, active/standby failover, singleton support.
14. Inbound API — Web API for external systems, API key auth, script-based implementations.
15. Host — Single deployable binary, role-based component registration, Akka.NET bootstrap.
16. Commons — Shared types, POCO entity classes, repository interfaces, message contracts.
17. Configuration Database — EF Core data access layer, repositories, unit-of-work, audit logging (IAuditService), migrations.
18. Management Service — Akka.NET actor providing programmatic access to all admin operations, ClusterClientReceptionist registration.
19. CLI — Command-line tool using HTTP Management API, System.CommandLine, JSON/table output.
20. Traefik Proxy — Reverse proxy/load balancer fronting central cluster, active node routing via `/health/active`, automatic failover.
21. Notification Outbox — Central component ingesting store-and-forwarded notifications, `Notifications` audit table, dispatcher loop, retry/parking, delivery KPIs.
22. Site Call Audit — Central component auditing site cached calls (`CachedCall`/`CachedWrite`); `SiteCalls` audit table, telemetry ingest, reconciliation, KPIs, central→site Retry/Discard relay; sites remain the source of truth.
23. Audit Log — Central append-only AuditLog table spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site SQLite hot-path + gRPC telemetry + reconciliation; combined telemetry with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API; monthly partitioning, 365-day retention.
24. Transport — File-based, encrypted bundle export/import via Central UI. Templates (with **all** child collections — attributes/alarms/scripts/compositions/**native-alarm-sources**, field-complete incl. `LockedInDerived` + script cadence/timeout), system artifacts, central-only configuration, plus site/instance-scoped config (`Site`s, site `DataConnection`s, `Instance`s + real `AreaName` by name) reconciled across environments by a `BundleNameMap` name-mapping subsystem. Per-conflict resolution with a per-line Myers diff. Runs the **script trust gate** at import review (5th `ScriptTrustValidator` call site — forbidden-API scripts rejected pre-runtime; covers template/shared/ApiMethod bodies, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** — all hard-block; template-script name-resolution findings are advisory warnings, ApiMethod findings hard-block). Correlated audit via `BundleImportId`. Never touches site runtime nodes (imported instances land `NotDeployed`).
25. Script Analysis — Shared authoritative script-trust analyzer: unified forbidden-API deny-list (`ScriptTrustPolicy`), fused semantic+syntactic validator (`ScriptTrustValidator`), Roslyn compile wrapper (`RoslynScriptCompiler`), and compile-only globals stubs (`ScriptCompileSurface`/`TriggerCompileSurface`); consumed by Template Engine, Site Runtime, Inbound API, and Central UI.
26. KPI History — Reusable central KPI-history backbone: tall/EAV `KpiSample` store in central MS SQL, `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated) sampling DI-registered `IKpiSampleSource`s every minute, bucketed query (`GetRawSeriesAsync` + `KpiSeriesBucketer`) + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart`; ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health.
27. DelmiaNotifier — Standalone external client tool (NOT a cluster component, NOT in the Host): a compact Native-AOT (`win-x64`) console app (`WWNotifier.exe`) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API `DelmiaRecipeDownload` method (`X-API-Key`, key from `SCADABRIDGE_API_KEY`), with connect-failure-only failover across a comma-list of base URLs, and reports the legacy `YES`/`NO` + exit-code stdout contract — a drop-in replacement for the legacy `WWNotifier` (see `docs/former-api-specs/dnc/`). Zero-dependency BCL-only, `System.Text.Json` source-gen. Project README: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`; design: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`.
## Key Design Decisions (for context across sessions) ## Key Design Decisions (for context across sessions)
@@ -120,30 +79,34 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- DCL write failures returned synchronously to calling script. - DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting. - Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing: - **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently. - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them: - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing. - CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node. - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`. - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB. - `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system. - All timestamps are UTC throughout the system.
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). - **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables. - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up). - Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
- **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
### External Integrations ### External Integrations
- External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth. - External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
- Dual call modes: `ExternalSystem.Call()` (synchronous) and `ExternalSystem.CachedCall()` (store-and-forward on transient failure). - Dual call modes: `ExternalSystem.Call()` (synchronous) and `ExternalSystem.CachedCall()` (store-and-forward on transient failure).
- Error classification: HTTP 5xx/408/429/connection errors = transient; other 4xx = permanent (returned to script). - Error classification: HTTP 5xx/408/429/connection errors = transient; other 4xx = permanent (returned to script).
- Notification Service: SMTP with OAuth2 Client Credentials (Microsoft 365) or Basic Auth. BCC delivery, plain text. - Notification Service: SMTP with OAuth2 Client Credentials (Microsoft 365) or Basic Auth. BCC delivery, plain text.
- Email delivery has **two transports**, selected per config row by `SmtpConfiguration.Transport` (null/`Smtp` = default): SMTP (MailKit, Basic/OAuth2) or on-prem Exchange **EWS** (no-SDK `CreateItem` SOAP over `HttpClient`, Basic-over-HTTPS only — https enforced at write gate, adapter AND sender — BCC-only recipients, `SendOnly` so no Sent-Items copy). Under `Ews`, `Host` is the full EWS URL, `Credentials` is `username:password`, and Port/TlsMode/OAuth2*/MaxConcurrentConnections are unused. Site-facing behavior is unchanged (delivery stays central-only). Q12 (M365 OAuth2 tenant) is closed as **superseded** — the OAuth2 SMTP path stays config-selectable but untested against a live tenant. Design: `docs/plans/2026-08-10-ews-email-transport-design.md`.
- Notification delivery is central-only: sites store-and-forward notifications to the central cluster (target = central, not SMTP); sites never talk to SMTP. Notification lists and SMTP config are no longer deployed to sites; recipient resolution happens at central, at delivery time. - Notification delivery is central-only: sites store-and-forward notifications to the central cluster (target = central, not SMTP); sites never talk to SMTP. Notification lists and SMTP config are no longer deployed to sites; recipient resolution happens at central, at delivery time.
- Notification lists carry a `Type` discriminator (`Email` and `Sms`). `Notify.To("list")` is type-agnostic; delivery is via per-type `INotificationDeliveryAdapter` (Email via SMTP; Sms via Twilio REST — `SmsNotificationDeliveryAdapter`, no SDK, one POST per recipient, per-recipient rollup). List Type is fixed after creation. - Notification lists carry a `Type` discriminator (`Email` and `Sms`). `Notify.To("list")` is type-agnostic; delivery is via per-type `INotificationDeliveryAdapter` (Email via SMTP; Sms via Twilio REST — `SmsNotificationDeliveryAdapter`, no SDK, one POST per recipient, per-recipient rollup). List Type is fixed after creation.
- `Notify.Send` is async and **enqueue-only** — it buffers the notification into the local SQLite S&F store and returns a `NotificationId` (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (`deferToSweep: true` buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with `maxRetries: 0` (the "no limit" escape hatch), so notifications retry until central acks and are **never parked for retry exhaustion** — only a corrupt payload parks them (arch-review 02, Tasks 13/14). `Notify.Status(notificationId)` returns a status record (status, retry count, last error, key timestamps); answered site-locally as `Forwarding` while still in the site S&F buffer, otherwise round-trips to central. - `Notify.Send` is async and **enqueue-only** — it buffers the notification into the local SQLite S&F store and returns a `NotificationId` (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (`deferToSweep: true` buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with `maxRetries: 0` (the "no limit" escape hatch), so notifications retry until central acks and are **never parked for retry exhaustion** — only a corrupt payload parks them (arch-review 02, Tasks 13/14). `Notify.Status(notificationId)` returns a status record (status, retry count, last error, key timestamps); answered site-locally as `Forwarding` while still in the site S&F buffer, otherwise round-trips to central.
@@ -188,7 +151,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Scope = script trust boundary: outbound API (sync + cached), outbound DB (sync + cached), notifications, inbound API. Framework/internal traffic is explicitly excluded. - Scope = script trust boundary: outbound API (sync + cached), outbound DB (sync + cached), notifications, inbound API. Framework/internal traffic is explicitly excluded.
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`). - One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots). - `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. Tag-cascade coverage is complete as of M5.4 (T4) — no further spawn points are deferred. - `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. - Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost.
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction. - Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7) - Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
@@ -208,16 +171,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`). - Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
### Cluster & Failover ### Cluster & Failover
- Keep-oldest split-brain resolver with `down-if-alone = on`, 15s stable-after. - **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- Both nodes are seed nodes. `min-nr-of-members = 1`. - Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s. - Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover. - CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage. - Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
### UI & Monitoring ### UI & Monitoring
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc. - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
- UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components. - UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR. - Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval. - Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
- Dead letter monitoring as a health metric. - Dead letter monitoring as a health metric.
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search. - Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
@@ -240,20 +206,15 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
### Akka.NET Conventions ### Akka.NET Conventions
- Tell for hot-path internal communication; Ask reserved for system boundaries. - Tell for hot-path internal communication; Ask reserved for system boundaries.
- ClusterClient for cross-cluster communication; ClusterClientReceptionist for service discovery across cluster boundaries. - Cross-cluster communication is gRPC (per-site PSK-authenticated): site→central `CentralControlService`, central→site `SiteCommandService`, plus the `SiteStreamService` data stream. ClusterClient/ClusterClientReceptionist were removed in the migration's Phase 4 — service discovery is by dialling configured endpoints, not the receptionist. (Akka.Cluster.Tools remains for ClusterSingleton.)
- Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — `ScriptTrustPolicy` is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to `ScriptTrustValidator`. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory. - Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — `ScriptTrustPolicy` is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to `ScriptTrustValidator`. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory.
- Application-level correlation IDs on all request/response messages. - Application-level correlation IDs on all request/response messages.
## Tool Usage ## Tool Usage
- When consulting with the Codex MCP tool, use model `gpt-5.4`.
- When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference. - When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.
### CLI Quick Reference (Docker / OrbStack) ### CLI Quick Reference (Docker / OrbStack)
- **Management URL**: `http://localhost:9000` — the CLI connects via the Traefik load balancer, which routes to the active central node. Direct access: central-a on port 9001, central-b on port 9002. Management URL, test credentials, rebuild/redeploy commands and the infra services are in the
- **Test user**: `--username multi-role --password password` — has Admin, Design, and Deployment roles. The `admin` user only has the Admin role and cannot create templates, data connections, or deploy. `scadabridge-cluster-ops` skill (`.claude/skills/scadabridge-cluster-ops/SKILL.md`).
- **Config file**: `~/.scadabridge/config.json` — stores `managementUrl` and default format. See `docker/README.md` for a ready-to-use test config.
- **Rebuild cluster**: `bash docker/deploy.sh` — builds the `scadabridge:latest` image and recreates all containers. Run this after code changes to ManagementActor, Host, or any server-side component.
- **Infrastructure services**: `cd infra && docker compose up -d` — starts MS SQL, OPC UA, SMTP, and REST API. These are separate from the cluster containers in `docker/`. **LDAP is NOT started here** — it is the shared GLAuth on `10.100.0.35:3893` (dc=zb,dc=local); source of truth and config: `scadaproj/infra/glauth/`.
- **All test LDAP passwords**: `password` (see `scadaproj/infra/glauth/config.toml` for users and groups; canonical cross-app login: `multi-role`).
Directory.Packages.props
+33 -11
View File
@@ -90,9 +90,9 @@
to mark tests as Skipped (not silently Passed) when MSSQL is unreachable. to mark tests as Skipped (not silently Passed) when MSSQL is unreachable.
--> -->
<PackageVersion Include="Xunit.SkippableFact" Version="1.5.61" /> <PackageVersion Include="Xunit.SkippableFact" Version="1.5.61" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Health" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
@@ -103,14 +103,15 @@
<PackageVersion Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.1.5" /> <PackageVersion Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" /> <PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" /> <PackageVersion Include="ZB.MOM.WW.Theme" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.1" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
</ItemGroup> </ItemGroup>
<!-- <!--
@@ -145,6 +146,27 @@
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" /> <PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
</ItemGroup> </ItemGroup>
<!--
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, pulled in TRANSITIVELY by
Microsoft.AspNetCore.DataProtection 10.0.7 (ConfigurationDatabase's DataProtection
key storage). With TreatWarningsAsErrors any FRESH restore — notably the docker
image build — went red (surfaced 2026-07-21; local builds had cached audit data).
Same pattern as SQLitePCLRaw above: pin the vulnerable transitive package to its
patched version (10.0.10) with an explicit <PackageReference> in the one project
where the chain enters (ConfigurationDatabase; every other resolver — AuditLog,
SiteCallAudit, Transport, PerformanceTests, tests — reaches it through that
ProjectReference). Bumping the DataProtection parent instead was tried and
rejected: 10.0.10 floors Microsoft.Extensions.* and (via the EFCore adapter)
Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump
(NU1605 downgrade errors) that belongs in its own reviewed commit.
-->
<ItemGroup>
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
</ItemGroup>
<!-- <!--
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
README.md
+2 -2
View File
@@ -16,7 +16,7 @@ This repository is the full **implementation** project for ScadaBridge — the C
| Central Database | MS SQL Server, Entity Framework Core | | Central Database | MS SQL Server, Entity Framework Core |
| Site Storage | SQLite (deployed configs, S&F buffer, event logs) | | Site Storage | SQLite (deployed configs, S&F buffer, event logs) |
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions | | Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365; SMS via Twilio REST); store-and-forwarded from sites | | Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
| Hosting | Windows Server, Windows Service | | Hosting | Windows Server, Windows Service |
| Cluster | Akka.NET Cluster (active/standby, keep-oldest SBR) | | Cluster | Akka.NET Cluster (active/standby, keep-oldest SBR) |
| Logging | Serilog (structured) | | Logging | Serilog (structured) |
@@ -83,7 +83,7 @@ Both stacks share the infrastructure services in [`infra/`](infra/) (MS SQL, LDA
| 5 | Central–Site Communication | [docs/requirements/Component-Communication.md](docs/requirements/Component-Communication.md) | Dual transport: Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data). 9 message patterns with per-pattern timeouts, SiteStreamGrpcServer/Client, application-level correlation IDs, transport heartbeat config, gRPC keepalive, message ordering, connection failure behavior. The gRPC stream additively carries the read-only native alarm mirror (computed + native OPC UA / MxAccess) via the enriched `AlarmStateUpdate`. | | 5 | Central–Site Communication | [docs/requirements/Component-Communication.md](docs/requirements/Component-Communication.md) | Dual transport: Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data). 9 message patterns with per-pattern timeouts, SiteStreamGrpcServer/Client, application-level correlation IDs, transport heartbeat config, gRPC keepalive, message ordering, connection failure behavior. The gRPC stream additively carries the read-only native alarm mirror (computed + native OPC UA / MxAccess) via the enriched `AlarmStateUpdate`. |
| 6 | Store-and-Forward Engine | [docs/requirements/Component-StoreAndForward.md](docs/requirements/Component-StoreAndForward.md) | Buffering (transient failures only), fixed-interval retry, parking, async best-effort replication, SQLite persistence at sites. | | 6 | Store-and-Forward Engine | [docs/requirements/Component-StoreAndForward.md](docs/requirements/Component-StoreAndForward.md) | Buffering (transient failures only), fixed-interval retry, parking, async best-effort replication, SQLite persistence at sites. |
| 7 | External System Gateway | [docs/requirements/Component-ExternalSystemGateway.md](docs/requirements/Component-ExternalSystemGateway.md) | HTTP/REST + JSON, API key/Basic Auth, per-system timeout, dual call modes (Call/CachedCall), transient/permanent error classification, dedicated blocking I/O dispatcher, ADO.NET connection pooling. | | 7 | External System Gateway | [docs/requirements/Component-ExternalSystemGateway.md](docs/requirements/Component-ExternalSystemGateway.md) | HTTP/REST + JSON, API key/Basic Auth, per-system timeout, dual call modes (Call/CachedCall), transient/permanent error classification, dedicated blocking I/O dispatcher, ADO.NET connection pooling. |
| 8 | Notification Service | [docs/requirements/Component-NotificationService.md](docs/requirements/Component-NotificationService.md) | Central-only — manages typed notification-list, SMTP, and SMS definitions; supplies per-type delivery adapters (Email via SMTP with OAuth2 (M365) or Basic Auth, BCC, plain text; SMS via Twilio REST, per-recipient, outbound-only); delivery performed by the Notification Outbox. | | 8 | Notification Service | [docs/requirements/Component-NotificationService.md](docs/requirements/Component-NotificationService.md) | Central-only — manages typed notification-list, SMTP, and SMS definitions; supplies per-type delivery adapters (Email via SMTP with OAuth2 (M365) or Basic Auth, or on-prem Exchange EWS, BCC, plain text; SMS via Twilio REST, per-recipient, outbound-only); delivery performed by the Notification Outbox. |
| 9 | Central UI | [docs/requirements/Component-CentralUI.md](docs/requirements/Component-CentralUI.md) | Blazor Server with SignalR real-time push, load balancer failover with JWT, all management workflows. Custom-content modal host (`DialogService.ShowAsync<T>`) with focus-trap/restore; dark-mode CSS-variable token layer (`[data-bs-theme="dark"]` overriding `ZB.MOM.WW.Theme` tokens in `site.css`, `localStorage`-persisted, SSR no-flash); reusable presentational components `OffsetPager`, `KeysetPager`, and `DateTimeRangeFilter` adopted across report/audit pages. | | 9 | Central UI | [docs/requirements/Component-CentralUI.md](docs/requirements/Component-CentralUI.md) | Blazor Server with SignalR real-time push, load balancer failover with JWT, all management workflows. Custom-content modal host (`DialogService.ShowAsync<T>`) with focus-trap/restore; dark-mode CSS-variable token layer (`[data-bs-theme="dark"]` overriding `ZB.MOM.WW.Theme` tokens in `site.css`, `localStorage`-persisted, SSR no-flash); reusable presentational components `OffsetPager`, `KeysetPager`, and `DateTimeRangeFilter` adopted across report/audit pages. |
| 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. | | 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. |
| 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. | | 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
ScadaBridge-docs-fixed.md
-116
View File
@@ -1,116 +0,0 @@
# Documentation Analysis Report
Files Scanned: 793
Files With Issues: 10
Total Issues: 11
## Issues
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/BrowserTime.cs
LINE: 26
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: UTC-5
MESSAGE: Comment contains 'UTC-5', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/TransportImport.razor.cs
LINE: 225
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: Step-2
MESSAGE: Comment contains 'Step-2', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/ConnectionHealthQueryService.cs
LINE: 18
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: PLC-1
MESSAGE: Comment contains 'PLC-1', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditExportHelpers.cs
LINE: 156
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: non-403
MESSAGE: Comment contains 'non-403', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/BundleCommands.cs
LINE: 407
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: non-403
MESSAGE: Comment contains 'non-403', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs
LINE: 43
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: sub-100
MESSAGE: Comment contains 'sub-100', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs
LINE: 29
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: year-0001
MESSAGE: Comment contains 'year-0001', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs
LINE: 2034
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs
LINE: 206
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationService/MailKitSmtpClientWrapper.cs
LINE: 9
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationService/MailKitSmtpClientWrapper.cs
LINE: 113
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
ScadaBridge-docs-issues.md
-4296
View File
File diff suppressed because it is too large Load Diff
alog.md
+9 -2
View File
@@ -108,7 +108,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `EventId` | `uniqueidentifier` PK | Generated where the event originates (site or central). Idempotency key. | | `EventId` | `uniqueidentifier` PK | Generated where the event originates (site or central). Idempotency key. |
| `OccurredAtUtc` | `datetime2` | When the event happened (call returned, retry attempted, etc.). | | `OccurredAtUtc` | `datetime2` | When the event happened (call returned, retry attempted, etc.). |
| `IngestedAtUtc` | `datetime2` | When central persisted the row (lags `OccurredAtUtc` for site-originated rows). | | `IngestedAtUtc` | `datetime2` | When central persisted the row (lags `OccurredAtUtc` for site-originated rows). |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound`. | | `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound` \| `SecuredWrite` \| `Cluster`. The last two are not script trust-boundary crossings: `SecuredWrite` records the two-person write lifecycle, and `Cluster` records operator-initiated topology actions (admin-triggered manual failover, decision 2026-07-22). |
| `Kind` | `varchar(32)` | Event kind discriminator (see kinds list below). | | `Kind` | `varchar(32)` | Event kind discriminator (see kinds list below). |
| `CorrelationId` | `uniqueidentifier` NULL | Ties multi-event operations together. `TrackedOperationId` for cached calls, `NotificationId` for notifications, request-id for inbound API. NULL for sync one-shot calls. | | `CorrelationId` | `uniqueidentifier` NULL | Ties multi-event operations together. `TrackedOperationId` for cached calls, `NotificationId` for notifications, request-id for inbound API. NULL for sync one-shot calls. |
| `SourceSiteId` | `varchar(64)` NULL | NULL for central-originated events (inbound API, central notification dispatch). | | `SourceSiteId` | `varchar(64)` NULL | NULL for central-originated events (inbound API, central notification dispatch). |
@@ -135,7 +135,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
- `IX_AuditLog_Target_Occurred (Target, OccurredAtUtc)` — "what did we send to system X." - `IX_AuditLog_Target_Occurred (Target, OccurredAtUtc)` — "what did we send to system X."
- Partitioning by month on `OccurredAtUtc` from day one (purge becomes a partition switch instead of a delete storm). - Partitioning by month on `OccurredAtUtc` from day one (purge becomes a partition switch instead of a delete storm).
**`Kind` values (flat — 10 discriminators across all channels):** **`Kind` values (flat — 17 discriminators across all channels; pinned by `AuditEnumTests`):**
| Kind | Fires when | | Kind | Fires when |
|---|---| |---|---|
@@ -149,6 +149,13 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `InboundAuthFailure` | An inbound API request was rejected at the auth boundary (bad/missing key). One row, `Status=Failed`, `HttpStatus=401`. | | `InboundAuthFailure` | An inbound API request was rejected at the auth boundary (bad/missing key). One row, `Status=Failed`, `HttpStatus=401`. |
| `CachedSubmit` | Script-side enqueue of a cached call (`ExternalSystem.CachedCall` / `Database.CachedWrite`); first row in the cached-call lifecycle, written to site SQLite before any forward attempt. | | `CachedSubmit` | Script-side enqueue of a cached call (`ExternalSystem.CachedCall` / `Database.CachedWrite`); first row in the cached-call lifecycle, written to site SQLite before any forward attempt. |
| `CachedResolve` | Terminal row for a cached operation — `Status` = `Delivered` / `Failed` / `Parked` / `Discarded`. | | `CachedResolve` | Terminal row for a cached operation — `Status` = `Delivered` / `Failed` / `Parked` / `Discarded`. |
| `SecuredWriteSubmit` | An Operator submitted a two-person secured write; row written after the `PendingSecuredWrite` is persisted so it carries the store-assigned id as `CorrelationId`. |
| `SecuredWriteApprove` | A distinct Verifier approved a pending secured write (no self-approval; enforced server-side). |
| `SecuredWriteReject` | A Verifier rejected a pending secured write. |
| `SecuredWriteExecute` | An approved secured write was relayed to the site MxGateway connection. |
| `SecuredWriteExpire` | A `Pending` secured write aged past its server-side TTL and was transitioned to `Expired` without executing — emitted by the system (no verifier). |
| `ReconciliationAbandoned` | A reconciliation pull row failed to insert up to the permanent-abandon threshold and central advanced its cursor past it; one synthetic row so the loss is queryable in the Audit Log itself. |
| `ManualFailover` | An administrator triggered a manual failover of the central pair from the Health page; one row per invocation, written BEFORE the graceful `Cluster.Leave` is issued. `Target` = the leaving node's address. |
### Site: `AuditLog` (SQLite) ### Site: `AuditLog` (SQLite)
archreview/plans/00-MASTER-TRACKER.md
+3 -1
View File
@@ -16,7 +16,7 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks
| Plan | Domain | Tasks | Done | Status | Findings coverage | | Plan | Domain | Tasks | Done | Status | Findings coverage |
|------|--------|------:|-----:|--------|-------------------| |------|--------|------:|-----:|--------|-------------------|
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 10 | ✅ Merged — T2 live-drill deferred | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 | | [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 11 | ✅ Merged — T2 RESOLVED 2026-08-01 (see note below) | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
| [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 15 | ✅ Merged | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 | | [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 15 | ✅ Merged | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 |
| [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 7 | ✅ Merged | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 | | [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 7 | ✅ Merged | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 |
| [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 13 | ✅ Merged | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 | | [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 13 | ✅ Merged | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 |
@@ -27,6 +27,8 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks
**Round-2 progress: COMPLETE — all 8 plans executed (TDD, per-task commits) and merged to `main` @ `1930f19b` on 2026-07-13 (fast-forward of the `r2-integration` assembly; origin pushed; PRs #6–#13 closed). ~81 of 86 tasks landed; the remainder are human/environment-gated: R2-08 T1 (rotate wonder-app-vd03 API key), T2/T11/T13 (delete untracked live-credential + generated files), and R2-01 T2 (live docker failover drill — in-process envelope already measured at 33.7s). Integration surfaced + fixed 2 cross-plan regressions no per-project run caught: R2-08 NodeName validator vs IntegrationTests host-boot, and R2-03 compile-cache counter under parallel test load.** **Round-2 progress: COMPLETE — all 8 plans executed (TDD, per-task commits) and merged to `main` @ `1930f19b` on 2026-07-13 (fast-forward of the `r2-integration` assembly; origin pushed; PRs #6–#13 closed). ~81 of 86 tasks landed; the remainder are human/environment-gated: R2-08 T1 (rotate wonder-app-vd03 API key), T2/T11/T13 (delete untracked live-credential + generated files), and R2-01 T2 (live docker failover drill — in-process envelope already measured at 33.7s). Integration surfaced + fixed 2 cross-plan regressions no per-project run caught: R2-08 NodeName validator vs IntegrationTests host-boot, and R2-03 compile-cache counter under parallel test load.**
> **Update 2026-08-07 (truth sweep):** the R2-01 T2 live failover drill is **RESOLVED as of 2026-08-01** — delivered via PLAN-R2-01 T4's live `FailoverTimingTests` on the real two-node in-process rig at production timings, plus `docker/failover-drill.sh` and `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs` (`AutoDown_HardCrashOfOldestNode_*`) covering the oldest-crash direction. Recorded in the deferred-work register (`docs/plans/2026-07-08-deferred-work-register.md`, "Failover-timing measurement" row, RESOLVED 2026-08-01). R2-08 T11 (delete root `deferred.md`) and T13 (delete generated root docs reports) were completed by this sweep (the files were in fact tracked, so they were `git rm`'d). Remaining human-gated residuals: R2-08 T1 (key rotation) and T2 (delete credential files).
## Round-2 P0 (do first, any order) ## Round-2 P0 (do first, any order)
1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`. 1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`.
archreview/plans/PLAN-R2-01-cluster-host-failover.md.tasks.json
+13 -12
View File
@@ -4,13 +4,13 @@
{ {
"id": 1, "id": 1,
"subject": "Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode", "subject": "Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 2, "id": 2,
"subject": "Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint", "subject": "Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
1 1
] ]
@@ -18,7 +18,8 @@
{ {
"id": 3, "id": 3,
"subject": "Task 3: RUN the drill live (both directions) and record measured timings", "subject": "Task 3: RUN the drill live (both directions) and record measured timings",
"status": "pending", "status": "completed",
"notes": "The live-drill residual (tracker: 'T2 live-drill deferred') was RESOLVED 2026-08-01 via PLAN-R2-01 T4's live FailoverTimingTests + docker/failover-drill.sh + SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*; recorded in the deferred-work register. Statuses in this manifest flipped from stale 'pending' per 00-MASTER-TRACKER.md by the 2026-08-07 truth sweep.",
"blockedBy": [ "blockedBy": [
1, 1,
2 2
@@ -27,25 +28,25 @@
{ {
"id": 4, "id": 4,
"subject": "Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process", "subject": "Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 5, "id": 5,
"subject": "Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)", "subject": "Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 6, "id": 6,
"subject": "Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step", "subject": "Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 7, "id": 7,
"subject": "Task 7: Correct the factually wrong N2 deferral record in the master tracker", "subject": "Task 7: Correct the factually wrong N2 deferral record in the master tracker",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
5, 5,
6 6
@@ -54,29 +55,29 @@
{ {
"id": 8, "id": 8,
"subject": "Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)", "subject": "Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 9, "id": 9,
"subject": "Task 9: Purge the stale \"cluster leader\" narration from CentralHealthReportLoop", "subject": "Task 9: Purge the stale \"cluster leader\" narration from CentralHealthReportLoop",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 10, "id": 10,
"subject": "Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope", "subject": "Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 11, "id": 11,
"subject": "Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains", "subject": "Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
10 10
] ]
} }
], ],
"lastUpdated": "2026-07-13T03:25:44Z" "lastUpdated": "2026-08-07T00:00:00Z"
} }
archreview/plans/PLAN-R2-02-communication-store-and-forward.md.tasks.json
+15 -15
View File
@@ -4,19 +4,19 @@
{ {
"id": 1, "id": 1,
"subject": "Task 1: Shared oldest-Up active-node evaluator, reachable from Communication and SiteRuntime", "subject": "Task 1: Shared oldest-Up active-node evaluator, reachable from Communication and SiteRuntime",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 2, "id": 2,
"subject": "Task 2: Two-node divergence repro — leader≠oldest wipes the delivering node's buffer (failing first)", "subject": "Task 2: Two-node divergence repro — leader≠oldest wipes the delivering node's buffer (failing first)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 3, "id": 3,
"subject": "Task 3: Swap SiteReplicationActor to the shared oldest-Up predicate + Host wires the delivery-gate delegate", "subject": "Task 3: Swap SiteReplicationActor to the shared oldest-Up predicate + Host wires the delivery-gate delegate",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
1, 1,
2 2
@@ -25,7 +25,7 @@
{ {
"id": 4, "id": 4,
"subject": "Task 4: Swap SiteCommunicationActor.DefaultIsActiveCheck + Host wiring + doc sync", "subject": "Task 4: Swap SiteCommunicationActor.DefaultIsActiveCheck + Host wiring + doc sync",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
3 3
] ]
@@ -33,7 +33,7 @@
{ {
"id": 5, "id": 5,
"subject": "Task 5: Chunked resync protocol — additive messages, byte-budgeted chunker, active-side chunked answer", "subject": "Task 5: Chunked resync protocol — additive messages, byte-budgeted chunker, active-side chunked answer",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
3 3
] ]
@@ -41,7 +41,7 @@
{ {
"id": 6, "id": 6,
"subject": "Task 6: Standby-side chunk assembly, atomic apply, ack + the N5 race comment", "subject": "Task 6: Standby-side chunk assembly, atomic apply, ack + the N5 race comment",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
5 5
] ]
@@ -49,7 +49,7 @@
{ {
"id": 7, "id": 7,
"subject": "Task 7: Resync delivery confirmation on the active node + telemetry + doc rewrite", "subject": "Task 7: Resync delivery confirmation on the active node + telemetry + doc rewrite",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
6 6
] ]
@@ -57,25 +57,25 @@
{ {
"id": 8, "id": 8,
"subject": "Task 8: Publish _sweepTask only when the sweep CAS is won (unclobber the shutdown drain)", "subject": "Task 8: Publish _sweepTask only when the sweep CAS is won (unclobber the shutdown drain)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 9, "id": 9,
"subject": "Task 9: Eager validation for SweepBatchLimit / SweepTargetParallelism", "subject": "Task 9: Eager validation for SweepBatchLimit / SweepTargetParallelism",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 10, "id": 10,
"subject": "Task 10: Coalesce live-alarm delta publishes (bound the per-circuit fan-out) [MUTEX with PLAN-R2-07]", "subject": "Task 10: Coalesce live-alarm delta publishes (bound the per-circuit fan-out) [MUTEX with PLAN-R2-07]",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 11, "id": 11,
"subject": "Task 11: Aggregator lifecycle — queued re-seed after reconnect + stream-generation stamp", "subject": "Task 11: Aggregator lifecycle — queued re-seed after reconnect + stream-generation stamp",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
10 10
] ]
@@ -83,7 +83,7 @@
{ {
"id": 12, "id": 12,
"subject": "Task 12: Instance-injected ReconnectDelay/StabilityWindow (kill the process-global test seams)", "subject": "Task 12: Instance-injected ReconnectDelay/StabilityWindow (kill the process-global test seams)",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
11 11
] ]
@@ -91,13 +91,13 @@
{ {
"id": 13, "id": 13,
"subject": "Task 13: Production caller for RemoveSiteAsync — dispose a deleted site's gRPC channels", "subject": "Task 13: Production caller for RemoveSiteAsync — dispose a deleted site's gRPC channels",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 14, "id": 14,
"subject": "Task 14: Documented acceptance — standby-node aggregators + deleted-site viewer behavior", "subject": "Task 14: Documented acceptance — standby-node aggregators + deleted-site viewer behavior",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
4 4
] ]
@@ -105,7 +105,7 @@
{ {
"id": 15, "id": 15,
"subject": "Task 15: Single-endpoint sites can go live [MUTEX with PLAN-R2-07]", "subject": "Task 15: Single-endpoint sites can go live [MUTEX with PLAN-R2-07]",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
12 12
] ]
archreview/plans/PLAN-R2-03-site-runtime-dcl.md.tasks.json
+7 -7
View File
@@ -1,13 +1,13 @@
{ {
"planPath": "archreview/plans/PLAN-R2-03-site-runtime-dcl.md", "planPath": "archreview/plans/PLAN-R2-03-site-runtime-dcl.md",
"tasks": [ "tasks": [
{ "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "pending", "blockedBy": [] }, { "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] }, { "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "completed", "blockedBy": [] },
{ "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] }, { "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "completed", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "pending", "blockedBy": [] }, { "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "completed", "blockedBy": [] },
{ "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "pending", "blockedBy": [] }, { "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "pending", "blockedBy": [5] }, { "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6] } { "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6] }
], ],
"lastUpdated": "2026-07-13T03:23:02Z" "lastUpdated": "2026-07-13T03:23:02Z"
} }
archreview/plans/PLAN-R2-04-data-audit-backbone.md.tasks.json
+13 -13
View File
@@ -1,19 +1,19 @@
{ {
"planPath": "archreview/plans/PLAN-R2-04-data-audit-backbone.md", "planPath": "archreview/plans/PLAN-R2-04-data-audit-backbone.md",
"tasks": [ "tasks": [
{ "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "pending", "blockedBy": [] }, { "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "pending", "blockedBy": [1] }, { "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "pending", "blockedBy": [2] }, { "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "completed", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "pending", "blockedBy": [2, 3] }, { "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "completed", "blockedBy": [2, 3] },
{ "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "pending", "blockedBy": [] }, { "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "pending", "blockedBy": [5] }, { "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "pending", "blockedBy": [6] }, { "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "completed", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "pending", "blockedBy": [] }, { "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "completed", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "pending", "blockedBy": [1, 2] }, { "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "completed", "blockedBy": [1, 2] },
{ "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "pending", "blockedBy": [] }, { "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "completed", "blockedBy": [] },
{ "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "pending", "blockedBy": [10] }, { "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "completed", "blockedBy": [10] },
{ "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "pending", "blockedBy": [] }, { "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "completed", "blockedBy": [] },
{ "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "pending", "blockedBy": [4, 7, 8, 9, 11, 12] } { "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "completed", "blockedBy": [4, 7, 8, 9, 11, 12] }
], ],
"lastUpdated": "2026-07-13T03:25:01Z" "lastUpdated": "2026-07-13T03:25:01Z"
} }
archreview/plans/PLAN-R2-05-templates-deployment-transport.md.tasks.json
+9 -9
View File
@@ -1,15 +1,15 @@
{ {
"planPath": "archreview/plans/PLAN-R2-05-templates-deployment-transport.md", "planPath": "archreview/plans/PLAN-R2-05-templates-deployment-transport.md",
"tasks": [ "tasks": [
{ "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "pending", "blockedBy": [] }, { "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "pending", "blockedBy": [1] }, { "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "pending", "blockedBy": [2] }, { "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "completed", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "pending", "blockedBy": [3] }, { "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "completed", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "pending", "blockedBy": [] }, { "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "pending", "blockedBy": [5] }, { "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "pending", "blockedBy": [6] }, { "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "completed", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "pending", "blockedBy": [] }, { "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "completed", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] } { "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] }
], ],
"lastUpdated": "2026-07-13T03:23:07Z" "lastUpdated": "2026-07-13T03:23:07Z"
} }
archreview/plans/PLAN-R2-06-edge-integrations.md.tasks.json
+6 -6
View File
@@ -1,12 +1,12 @@
{ {
"planPath": "archreview/plans/PLAN-R2-06-edge-integrations.md", "planPath": "archreview/plans/PLAN-R2-06-edge-integrations.md",
"tasks": [ "tasks": [
{ "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "pending", "blockedBy": [] }, { "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "pending", "blockedBy": [1] }, { "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "pending", "blockedBy": [] }, { "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "completed", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "pending", "blockedBy": [3] }, { "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "completed", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "pending", "blockedBy": [] }, { "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "pending", "blockedBy": [] } { "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "completed", "blockedBy": [] }
], ],
"lastUpdated": "2026-07-13T03:23:07Z" "lastUpdated": "2026-07-13T03:23:07Z"
} }
archreview/plans/PLAN-R2-07-ui-management-security.md.tasks.json
+12 -12
View File
@@ -4,19 +4,19 @@
{ {
"id": 1, "id": 1,
"subject": "Task 1: Route the DebugStreamHub LDAP bind through ManagementAuthenticator (throttled) + fix the false doc claim", "subject": "Task 1: Route the DebugStreamHub LDAP bind through ManagementAuthenticator (throttled) + fix the false doc claim",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 2, "id": 2,
"subject": "Task 2: ForwardedHeadersSetup — trusted-proxy client-IP resolution for the throttle keys", "subject": "Task 2: ForwardedHeadersSetup — trusted-proxy client-IP resolution for the throttle keys",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 3, "id": 3,
"subject": "Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off", "subject": "Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
2 2
] ]
@@ -24,19 +24,19 @@
{ {
"id": 4, "id": 4,
"subject": "Task 4: Enforce site scope on secured-write submit / approve / reject", "subject": "Task 4: Enforce site scope on secured-write submit / approve / reject",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 5, "id": 5,
"subject": "Task 5: ISecuredWriteRepository — additive permitted-sites filter for scoped listing", "subject": "Task 5: ISecuredWriteRepository — additive permitted-sites filter for scoped listing",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 6, "id": 6,
"subject": "Task 6: Scope-filter HandleListSecuredWrites + amend Component-Security.md:141 + matrix verification", "subject": "Task 6: Scope-filter HandleListSecuredWrites + amend Component-Security.md:141 + matrix verification",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
4, 4,
5 5
@@ -45,13 +45,13 @@
{ {
"id": 7, "id": 7,
"subject": "Task 7: AlarmSummary.RefreshAsync stale-site guard — never display cross-site data", "subject": "Task 7: AlarmSummary.RefreshAsync stale-site guard — never display cross-site data",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 8, "id": 8,
"subject": "Task 8: While live, the poll updates only _notReporting — never regresses live rows", "subject": "Task 8: While live, the poll updates only _notReporting — never regresses live rows",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
7 7
] ]
@@ -59,7 +59,7 @@
{ {
"id": 9, "id": 9,
"subject": "Task 9: House disposal guard on the live callback", "subject": "Task 9: House disposal guard on the live callback",
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
8 8
] ]
@@ -67,19 +67,19 @@
{ {
"id": 10, "id": 10,
"subject": "Task 10: Un-stick IsLive — deathwatch resets liveness when the aggregator terminates (MUTEX with PLAN-R2-02 SiteAlarmLiveCacheService task)", "subject": "Task 10: Un-stick IsLive — deathwatch resets liveness when the aggregator terminates (MUTEX with PLAN-R2-02 SiteAlarmLiveCacheService task)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 11, "id": 11,
"subject": "Task 11: Amortize LoginThrottle.Prune off the failure hot path", "subject": "Task 11: Amortize LoginThrottle.Prune off the failure hot path",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 12, "id": 12,
"subject": "Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation", "subject": "Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
} }
], ],
archreview/plans/PLAN-R2-08-conventions-hygiene.md.tasks.json
+20 -16
View File
@@ -5,80 +5,84 @@
"id": 1, "id": 1,
"subject": "Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — needs-user", "subject": "Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — needs-user",
"status": "pending", "status": "pending",
"blockedBy": [] "blockedBy": [],
"notes": "Still open per 00-MASTER-TRACKER.md (human-gated residual; needs-user)."
}, },
{ {
"id": 2, "id": 2,
"subject": "Task 2: Delete test.txt and triage the root credential files — needs-user", "subject": "Task 2: Delete test.txt and triage the root credential files — needs-user",
"status": "pending", "status": "pending",
"blockedBy": [1] "blockedBy": [1],
"notes": "Still open per 00-MASTER-TRACKER.md (human-gated residual; needs-user)."
}, },
{ {
"id": 3, "id": 3,
"subject": "Task 3: Root secret-capture guards — .gitignore patterns + pre-commit secret scan", "subject": "Task 3: Root secret-capture guards — .gitignore patterns + pre-commit secret scan",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 4, "id": 4,
"subject": "Task 4: CHANGELOG.md — correct the false role claim and refresh to reality", "subject": "Task 4: CHANGELOG.md — correct the false role claim and refresh to reality",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 5, "id": 5,
"subject": "Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)", "subject": "Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 6, "id": 6,
"subject": "Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)", "subject": "Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)",
"status": "pending", "status": "completed",
"blockedBy": [5] "blockedBy": [5]
}, },
{ {
"id": 7, "id": 7,
"subject": "Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)", "subject": "Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 8, "id": 8,
"subject": "Task 8: OperationTrackingOptions binding + the missing site IOperationTrackingStore registration (verify-then-fix)", "subject": "Task 8: OperationTrackingOptions binding + the missing site IOperationTrackingStore registration (verify-then-fix)",
"status": "pending", "status": "completed",
"blockedBy": [7] "blockedBy": [7]
}, },
{ {
"id": 9, "id": 9,
"subject": "Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings", "subject": "Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings",
"status": "pending", "status": "completed",
"blockedBy": [8] "blockedBy": [8]
}, },
{ {
"id": 10, "id": 10,
"subject": "Task 10: Re-consolidate deferral tracking into the canonical register", "subject": "Task 10: Re-consolidate deferral tracking into the canonical register",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 11, "id": 11,
"subject": "Task 11: Delete the drifted root deferred.md snapshot — needs-user", "subject": "Task 11: Delete the drifted root deferred.md snapshot — needs-user",
"status": "pending", "status": "completed",
"blockedBy": [10] "blockedBy": [10],
"notes": "Completed 2026-08-07 (truth sweep): deferred.md was in fact TRACKED (the plan's 'untracked' premise was wrong), so it was removed via git rm."
}, },
{ {
"id": 12, "id": 12,
"subject": "Task 12: Fix the stale Transport area comment in EntitySerializer.FromBundleContent", "subject": "Task 12: Fix the stale Transport area comment in EntitySerializer.FromBundleContent",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": []
}, },
{ {
"id": 13, "id": 13,
"subject": "Task 13: Commit the orphaned MES plan doc; remove the generated root reports — needs-user (deletions only)", "subject": "Task 13: Commit the orphaned MES plan doc; remove the generated root reports — needs-user (deletions only)",
"status": "pending", "status": "completed",
"blockedBy": [] "blockedBy": [],
"notes": "Completed 2026-08-07 (truth sweep): ScadaBridge-docs-issues.md + ScadaBridge-docs-fixed.md were in fact TRACKED and removed via git rm; the MES plan doc was already committed earlier (3c9b101d). Other tasks in this manifest flipped to completed per 00-MASTER-TRACKER.md (round 2 merged to main @ 1930f19b, 2026-07-13)."
} }
], ],
"lastUpdated": "2026-07-13T03:30:22Z" "lastUpdated": "2026-08-07T00:00:00Z"
} }
deferred.md
-44
View File
@@ -1,44 +0,0 @@
# Remaining Deferred Work
Source: `docs/plans/2026-07-08-deferred-work-register.md` (snapshot 2026-07-10).
Everything in the register's "Fix-now" table is already landed via the archreview
plans; what's left are the intentional deferrals below.
## Product / roadmap-locked (revisit needs a decision or a trigger event)
| # | Item | Why deferred | Revisit trigger |
|---|------|--------------|-----------------|
| 8 | Hash-chain tamper evidence (CLI `verify-chain` is a no-op stub) | v1.x by locked decision — append-only DB roles are the current control | A compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (endpoint returns `501`) | v1.x — the `501` + CLI messaging are honest, not broken | AuditLog partition volume nears the retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | Broadcast-to-both-nodes already covers HA | A governance/audit requirement for trust decisions |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | v1 manifest hash + AES-GCM held sufficient | A non-repudiation requirement across orgs |
| 17 | Unified notifications + site-calls outbox page | Explicit M9 decision to keep two pages | Operator confusion reports |
| 18 | Folder drag-drop | Permanently closed — menu reorder shipped instead | — (closed) |
## Scale / YAGNI (deferred until load justifies it)
| # | Item | Why deferred | Revisit trigger | Status |
|---|------|--------------|-----------------|--------|
| 10 | Aggregated live alarm stream for Alarm Summary | Snapshot fan-out is acceptable at current instance counts | Latency complaints or >~50 instances/site | ✅ **SHIPPED + MERGED to main 2026-07-10** (`8c888f13`, plan `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`, T1–T8). Transient per-site in-memory live cache (`ISiteAlarmLiveCache`/`SiteAlarmAggregatorActor`) seeded by snapshot fan-out + additive `SubscribeSite` alarm-only gRPC stream; live-cache-driven Alarm Summary with 15s poll fallback; `[PERM]` no-central-store honored (code-reviewer-confirmed); validated options + telemetry; end-to-end trace. Register row moved to Resolved. |
| 22 | KPI history hourly rollups | 90-day retention already bounds the table | `KpiSample` query latency on dashboards | ✅ **SHIPPED + MERGED to main 2026-07-10** (`8c888f13`, plan `docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8). Separate `KpiRollupHourly` table (migration `20260710153953`), recorder hourly fold w/ failover-safe lookback re-fold + idempotent upsert, per-metric gauge-vs-rate aggregation, range-threshold query routing (`RollupThresholdHours` 168h), longer rollup retention (365 ≥ 90) + dual purge, one-shot backfill, and 30 d/90 d window buttons. Register row moved to Resolved. |
## Low-priority polish (near-complete, small remainder)
| # | Item | Why deferred | Revisit trigger |
|---|------|--------------|-----------------|
| 12 | Native-alarm-source-override CSV import — Central UI upload button only | CLI + Management API + parser shipped 2026-07-10; the Blazor upload affordance is the only piece left, and it's pure polish (needs a live Blazor smoke) | First request to bulk-import native sources from the UI instead of the CLI |
## New deferrals from review 08 (engineering debt, no defect)
| Item | Why deferred | Revisit trigger |
|------|--------------|-----------------|
| Communication → HealthMonitoring layering inversion | Moving the interface + `SiteHealthState` to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to `ICentralHealthAggregator` |
| Reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Full StyleGuide-conformant docs are substantial; README claim was scoped instead (PLAN-08 T10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests, DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | Needs the PLAN-01 two-node rig; placeholder harness already shipped (PLAN-08 T8) | PLAN-01 rig landing |
---
Summary: 12 open deferrals (13th, folder drag-drop #18, is permanently closed).
None are currently actionable without a triggering event or product decision —
except row #12's UI upload button, whose CLI/API/parser core already shipped.
docker-env2/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", "akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" "akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
docker-env2/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", "akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" "akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
docker-env2/docker-compose.yml
+8
View File
@@ -16,6 +16,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a # pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it. # true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001" ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports: ports:
- "9101:5000" # Web UI + Inbound API - "9101:5000" # Web UI + Inbound API
- "9111:8081" # Akka remoting - "9111:8081" # Akka remoting
@@ -43,6 +47,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a # pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it. # true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001" ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports: ports:
- "9102:5000" # Web UI + Inbound API - "9102:5000" # Web UI + Inbound API
- "9112:8081" # Akka remoting - "9112:8081" # Akka remoting
docker-env2/site-x-node-a/appsettings.Site.json
+11 -4
View File
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082", "akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082" "akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
@@ -40,9 +40,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker-env2/site-x-node-b/appsettings.Site.json
+13 -6
View File
@@ -10,10 +10,10 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082", "akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082" "akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
@@ -40,9 +40,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/README.md
+66 -12
View File
@@ -120,6 +120,31 @@ docker/
└── logs/ └── logs/
``` ```
## gRPC control-plane keys (dev)
The site gRPC service (`SiteStreamService` on 8083 — live subscriptions, audit pull,
cached-telemetry ingest) is gated by a preshared key, and the gate is **fail-closed**: a site node
with no key refuses every call, and `StartupValidator` refuses to boot it at all. So the rig
carries dev keys, one per site:
| Where | Setting | Value |
|---|---|---|
| `site-{a,b,c}-node-*/appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | `dev-grpc-psk-docker-site-{a,b,c}` |
| `docker-compose.yml`, both central nodes | `ScadaBridge__Communication__SitePsks__site-{a,b,c}` | same value |
Both nodes of a pair carry the same key; each site's key is different from the others'. The
central half lives in compose env rather than the mounted `appsettings.Central.json`, which by
convention holds no plaintext credentials. Production uses `${secret:SB-GRPC-PSK-<siteId>}` on
the site and the matching secret in central's store — see
[`docs/deployment/topology-guide.md`](../docs/deployment/topology-guide.md).
**These are not real secrets and are committed deliberately**, exactly like the LocalDb sync key
(`dev-site-a-localdb-sync-key`) beside them. The two are separate keys on purpose: the LocalDb one
authenticates the *pair partner* for database replication, not central.
If you add a site to the rig, add its key in both places or its streams will fail with
`PermissionDenied`.
## Commands ## Commands
### Initial Setup ### Initial Setup
@@ -273,29 +298,43 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
### Automated failover drill (`failover-drill.sh`) ### Automated failover drill (`failover-drill.sh`)
```bash ```bash
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER
``` ```
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive: The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up. - **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered. - **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** The survivor must **take over while the victim is still down**: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its `/health/active` goes 200. PASS = survivor active within `TIMEOUT_S`, then Traefik routing to it. (Under the pre-2026-07-21 `keep-oldest` strategy this direction was a proven total outage — the younger survivor took `DownReachable` and downed itself, because Akka's `down-if-alone` only rescues a side with ≥ 2 members.)
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host. Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user). **Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
> **Observed results** (plan R2-01 T3): **Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
> **Observed results** (auto-down decision verification):
> >
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts. > **Run 2026-07-21** against a freshly-deployed cluster with `SplitBrainResolverStrategy: auto-down` (first drill: `active=central-a`). Both directions recovered.
> >
> | Direction (`DRILL_MODE`) | Outcome | Measured | > | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------| > |--------------------------|---------|----------|
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. | > | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. | > | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
> >
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**. > Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
### Central Failover ### Central Failover
@@ -313,6 +352,14 @@ open http://localhost:9002
docker start scadabridge-central-a docker start scadabridge-central-a
``` ```
**Manual failover from the UI (admin-only).** Instead of stopping a container, an Administrator can trigger a planned role swap from the **Trigger failover** button on the central-cluster card at `/monitoring/health` (via Traefik, `http://localhost:9000`). The active (oldest Up) node leaves the cluster **gracefully**, so singletons hand over rather than being killed; the node then restarts under `restart: unless-stopped` and rejoins as the standby.
- The button is disabled when the pair has no online standby — the same guard is re-enforced server-side, since failing over a lone node is an outage, not a failover.
- Triggering it **disconnects the page you clicked it on**: Traefik routes the UI to the active node, which is the node being restarted. The page reconnects against the new active node.
- Each invocation writes one `Cluster` / `ManualFailover` row to `dbo.AuditLog` naming the admin and the target address, written before the Leave is issued.
To verify on the rig: press the button, watch `central-a` restart and `central-b`'s badge flip to Primary, then confirm the audit row landed.
### Site Failover ### Site Failover
```bash ```bash
@@ -329,3 +376,10 @@ docker start scadabridge-site-a-a
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`). Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver). Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.
docker/central-node-a/appsettings.Central.json
+9 -2
View File
@@ -11,11 +11,18 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)." "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
docker/central-node-b/appsettings.Central.json
+11 -4
View File
@@ -8,14 +8,21 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-a:8081"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)." "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
docker/docker-compose.yml
+115
View File
@@ -1,12 +1,81 @@
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
#
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
# the wire, so every participating node must resolve the SAME KEK.
#
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
# site-b and site-c are deliberately left without it, so the default-OFF posture is
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
# pair. A node with no Secrets__* override keeps the shipped appsettings default
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
#
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
# since resolving one is what the hub exists to make possible.
x-secrets-hub-env: &secrets-hub-env
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
# kek_id mismatch that reads like corruption but is a deployment error.
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
# The appsettings default is the relative "scadabridge-secrets.db", which resolves to
# /app — inside the image's writable layer, so it is destroyed by any container
# recreate and unreachable from the host. /app/data is the node's own mounted volume
# (the one LocalDb already uses on sites; added to the central pair for this).
Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__Mode: "Grpc"
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
# hub and every follower; an unset token is a startup failure on both halves.
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
# so the two hub instances serve identical manifests by construction — that is what makes
# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
# reach the store that would resolve it (bootstrap circularity; registration rejects it).
# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
x-secrets-hub-central-env: &secrets-hub-central-env
Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# Site half of the same section. The hub client dials Endpoint first and fails over, per
# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
# above — never list endpoints backed by independent stores: failing over to an emptier
# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
# exists to prevent.
x-secrets-hub-site-env: &secrets-hub-site-env
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
services: services:
central-a: central-a:
image: scadabridge:latest image: scadabridge:latest
# An init process (tini) as PID 1, so a crashed dotnet process actually dies.
# Without it dotnet IS PID 1, Linux ignores the SIGABRT the runtime's crash path
# raises against PID 1, and any unhandled boot exception left the container
# `running` with the main thread spinning at 100% CPU — restart policy never
# fired (ScadaBridge#34). Belt to Program.cs's UnhandledException handler, which
# covers managed exceptions but not FailFast/runtime-internal aborts.
init: true
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s stop_grace_period: 30s
container_name: scadabridge-central-a container_name: scadabridge-central-a
environment: environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000" ASPNETCORE_URLS: "http://+:5000"
@@ -27,11 +96,26 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true" ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123" ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long" ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports: ports:
- "9001:5000" # Web UI + Inbound API - "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging) - "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes: volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-a/data:/app/data
- ./central-node-a/logs:/app/logs - ./central-node-a/logs:/app/logs
networks: networks:
- scadabridge-net - scadabridge-net
@@ -39,12 +123,16 @@ services:
central-b: central-b:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s stop_grace_period: 30s
container_name: scadabridge-central-b container_name: scadabridge-central-b
environment: environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000" ASPNETCORE_URLS: "http://+:5000"
@@ -65,11 +153,26 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true" ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123" ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long" ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports: ports:
- "9002:5000" # Web UI + Inbound API - "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting - "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes: volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-b/data:/app/data
- ./central-node-b/logs:/app/logs - ./central-node-b/logs:/app/logs
networks: networks:
- scadabridge-net - scadabridge-net
@@ -77,12 +180,16 @@ services:
site-a-a: site-a-a:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s stop_grace_period: 30s
container_name: scadabridge-site-a-a container_name: scadabridge-site-a-a
environment: environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site SCADABRIDGE_CONFIG: Site
ports: ports:
- "9021:8082" # Akka remoting (host access for debugging) - "9021:8082" # Akka remoting (host access for debugging)
@@ -97,12 +204,16 @@ services:
site-a-b: site-a-b:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s stop_grace_period: 30s
container_name: scadabridge-site-a-b container_name: scadabridge-site-a-b
environment: environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site SCADABRIDGE_CONFIG: Site
ports: ports:
- "9022:8082" # Akka remoting - "9022:8082" # Akka remoting
@@ -117,6 +228,7 @@ services:
site-b-a: site-b-a:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -137,6 +249,7 @@ services:
site-b-b: site-b-b:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -157,6 +270,7 @@ services:
site-c-a: site-c-a:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -177,6 +291,7 @@ services:
site-c-b: site-c-b:
image: scadabridge:latest image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting + # CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed # actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]). # mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
docker/failover-drill.sh
+52 -39
View File
@@ -1,33 +1,33 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first). # Failover drill against the running docker cluster (bash docker/deploy.sh first).
# #
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the # AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
# ACTIVE central node — but under the unified oldest-member semantics the # downing strategy (availability-first): the leader among the REACHABLE members
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT # downs the unreachable peer after StableAfter, so a hard crash of EITHER
# survive (registered deferred user decision, master tracker 2026-07-08; # central node — the active/oldest included — fails over to the survivor. The
# SbrFailoverTests.cs XML doc). Two modes: # accepted trade (made explicitly by the owner) is dual-active during a real
# network partition. Both drill directions therefore expect RECOVERY:
# #
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node. # DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node # The active node is untouched: expect zero /health/active routing blips
# keeps its singletons, and Traefik routing never goes dark. PASS = the # and member removal on the survivor within ~25s (10s failure detection +
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s # 15s auto-down-unreachable-after).
# failure detection + 15s stable-after) while /health/active stays up.
# #
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED # DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition # MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
# without the oldest, so the younger survivor downs ITSELF (down-if-alone # the singletons, and /health/active goes 200 on the survivor WHILE THE
# cannot help — the alone-oldest is dead and cannot down itself), and the # VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
# self-downed survivor cannot re-form a cluster alone unless it is the # margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
# FIRST seed (both nodes list central-a first; only the first seed may # was a total outage — the younger survivor downed ITSELF, verified live;
# self-join). This mode measures the dark window and PASSes only when # Akka's down-if-alone only rescues a side with >= 2 members.)
# central recovers AFTER the victim container is restarted. It exists to #
# make the registered gap observable — not to pretend it is covered. # Both modes finish by restarting the victim and confirming it rejoins as a
# fresh incarnation (standby).
set -euo pipefail set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}" TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}" TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}" DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() { active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
@@ -35,6 +35,7 @@ active_container() {
else echo "ERROR: no active central node found" >&2; exit 1; fi else echo "ERROR: no active central node found" >&2; exit 1; fi
} }
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; } peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
case "$DRILL_MODE" in case "$DRILL_MODE" in
standby|active) ;; standby|active) ;;
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
else else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE") VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi fi
SURVIVOR_PORT=$(port_of "$SURVIVOR")
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}" echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
START=$(date +%s) START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..." echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
BLIPS=0 BLIPS=0
while true; do while true; do
ELAPSED=$(( $(date +%s) - START )) ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1)) curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)." echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)." echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break break
fi fi
if (( ELAPSED > TIMEOUT_S )); then if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2 echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
docker start "${VICTIM}" > /dev/null docker start "${VICTIM}" > /dev/null
exit 1 exit 1
fi fi
sleep 1 sleep 1
done done
else else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..." echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
DARK_STREAK=0
while true; do while true; do
ELAPSED=$(( $(date +%s) - START )) ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
if (( DARK_STREAK >= 10 )); then echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed" echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
break break
fi fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then if (( ELAPSED > TIMEOUT_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the" echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it." docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
break docker start "${VICTIM}" > /dev/null
exit 1
fi fi
sleep 1 sleep 1
done done
echo "Confirming Traefik routes to the new active node..."
TR_START=$(date +%s)
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
if (( $(date +%s) - TR_START > 60 )); then
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
fi fi
echo "Restarting ${VICTIM}..." echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s) RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..." echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
VICTIM_PORT=$(port_of "$VICTIM")
while true; do while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT )) ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart." echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
break break
fi fi
if (( ELAPSED > 120 )); then if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2 echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
exit 1 exit 1
fi fi
sleep 1 sleep 1
done done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):" echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary." echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
docker/regen-proto.sh
+88 -49
View File
@@ -1,92 +1,131 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# #
# Regenerates the gRPC C# files from sitestream.proto. # Regenerates the gRPC C# files from the Communication project's .proto files.
# #
# Background: protoc (linux/arm64) segfaults inside our Docker build container # Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated Sitestream.cs + # (Grpc.Tools). As a workaround the generated C# is checked into
# SitestreamGrpc.cs are checked into src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/ # src/ZB.MOM.WW.ScadaBridge.Communication/ — SiteStreamGrpc/ for sitestream.proto,
# and the Protobuf ItemGroup in the .csproj is commented out — Docker just # CentralControlGrpc/ for central_control.proto, and SiteCommandGrpc/ for
# compiles the checked-in C# files. # site_command.proto — and the Protobuf ItemGroup in the .csproj is commented out,
# so Docker just compiles the checked-in files.
# #
# Run this script ON YOUR DEV MACHINE whenever Protos/sitestream.proto changes: # Run this script ON YOUR DEV MACHINE whenever a .proto changes:
# #
# 1. Temporarily uncomments the Protobuf ItemGroup so Grpc.Tools runs. # docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all] (default: all)
# 2. dotnet build (regen writes fresh files to obj/). #
# 3. Copies the regenerated files back into SiteStreamGrpc/. # 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 4. Re-comments the Protobuf ItemGroup so Docker builds stay safe. # 2. Deletes the stale checked-in C# so a failed regen is obvious.
# 3. dotnet build (regen writes fresh files to obj/).
# 4. Copies the regenerated files back into the source tree.
# 5. Restores the original csproj so no active Protobuf item is left behind.
#
# Only the SELECTED protos get a Protobuf item. Enabling one whose generated C#
# is still checked in would define every generated type twice, which is why the
# per-proto selection exists. central_control.proto imports sitestream.proto,
# but protoc resolves that from the project-relative path — the import needs no
# Protobuf item of its own.
# #
# Once we move to a Dockerfile base image that ships a working linux/arm64 # Once we move to a Dockerfile base image that ships a working linux/arm64
# protoc, this script can be retired and Docker can regen the proto on every # protoc, this script can be retired and Docker can regen the protos on every
# build like every other normal .NET project. # build like every other normal .NET project.
set -euo pipefail set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|sitecommand|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|sitecommand|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication" COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication"
CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj" CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj"
GEN_DIR="$COMM_DIR/SiteStreamGrpc" GEN="$COMM_DIR/obj/Debug/net10.0/Protos"
echo "=== Regenerating gRPC files from sitestream.proto ===" echo "=== Regenerating gRPC files ($TARGET) ==="
if [[ ! -f "$CSPROJ" ]]; then if [[ ! -f "$CSPROJ" ]]; then
echo "ERROR: csproj not found at $CSPROJ" >&2 echo "ERROR: csproj not found at $CSPROJ" >&2
exit 1 exit 1
fi fi
# Backup so we can always restore the comment state on failure. # Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
BACKUP="$(mktemp)" BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP" cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
# 1. Uncomment the Protobuf ItemGroup (strip the surrounding <!-- ... --> wrapper). # 1. Inject an ItemGroup holding just the selected protos, immediately before
python3 - <<PY # the closing </Project>. The documented commented-out block is left alone.
import re, pathlib python3 - "$CSPROJ" "$TARGET" <<'PY'
p = pathlib.Path("$CSPROJ") import pathlib, sys
src = p.read_text()
# Find the commented Protobuf block and unwrap it. csproj, target = pathlib.Path(sys.argv[1]), sys.argv[2]
new = re.sub(
r"<!--\s*\n(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)\s*\n\s*-->", protos = []
r"\1", if target in ("sitestream", "all"):
src, protos.append("sitestream.proto")
count=1, if target in ("centralcontrol", "all"):
) protos.append("central_control.proto")
if new == src: if target in ("sitecommand", "all"):
raise SystemExit("Couldn't find commented Protobuf ItemGroup to enable.") protos.append("site_command.proto")
p.write_text(new)
items = "\n".join(
f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
block = f" <ItemGroup>\n{items}\n </ItemGroup>\n\n</Project>"
src = csproj.read_text()
if "</Project>" not in src:
raise SystemExit("Couldn't find </Project> to inject the Protobuf ItemGroup before.")
csproj.write_text(src.replace("</Project>", block, 1))
PY PY
# 2. Delete the stale files so any failure to regen is obvious. # 2. Delete the stale files so any failure to regen is obvious.
rm -f "$GEN_DIR/Sitestream.cs" "$GEN_DIR/SitestreamGrpc.cs" if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteStreamGrpc/Sitestream.cs" "$COMM_DIR/SiteStreamGrpc/SitestreamGrpc.cs"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteCommandGrpc/SiteCommand.cs" \
"$COMM_DIR/SiteCommandGrpc/SiteCommandGrpc.cs"
fi
# 3. Regenerate by building. # 3. Regenerate by building.
echo "Building Communication project (regen)..." echo "Building Communication project (regen)..."
dotnet build "$CSPROJ" --nologo -v minimal | tail -5 dotnet build "$CSPROJ" --nologo -v minimal | tail -5
# 4. Copy generated files back into the source tree. # 4. Copy generated files back into the source tree.
mkdir -p "$GEN_DIR" if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
cp "$COMM_DIR/obj/Debug/net10.0/Protos/Sitestream.cs" "$GEN_DIR/Sitestream.cs" mkdir -p "$COMM_DIR/SiteStreamGrpc"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/SitestreamGrpc.cs" "$GEN_DIR/SitestreamGrpc.cs" cp "$GEN/Sitestream.cs" "$GEN/SitestreamGrpc.cs" "$COMM_DIR/SiteStreamGrpc/"
echo "Copied regenerated files to $GEN_DIR/" echo "Copied regenerated files to SiteStreamGrpc/"
fi
# 5. Re-comment the Protobuf ItemGroup so Docker builds keep working. if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
python3 - <<PY mkdir -p "$COMM_DIR/CentralControlGrpc"
import re, pathlib cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
p = pathlib.Path("$CSPROJ") echo "Copied regenerated files to CentralControlGrpc/"
src = p.read_text() fi
new = re.sub( if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
r"(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)", mkdir -p "$COMM_DIR/SiteCommandGrpc"
r"\n <!--\1\n -->", cp "$GEN/SiteCommand.cs" "$GEN/SiteCommandGrpc.cs" "$COMM_DIR/SiteCommandGrpc/"
src, echo "Copied regenerated files to SiteCommandGrpc/"
count=1, fi
)
p.write_text(new)
PY
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP" rm -f "$BACKUP"
trap - ERR trap - ERR
echo "" echo ""
echo "Done. Review and commit:" echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto" echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/" echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
docker/site-a-node-a/appsettings.Site.json
+19 -5
View File
@@ -14,11 +14,18 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082", "akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082" "akka.tcp://scadabridge@scadabridge-site-a-b:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/site-a-node-b/appsettings.Site.json
+21 -7
View File
@@ -11,14 +11,21 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-a-a:8082", "akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082" "akka.tcp://scadabridge@scadabridge-site-a-a:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/site-b-node-a/appsettings.Site.json
+19 -5
View File
@@ -14,11 +14,18 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082", "akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082" "akka.tcp://scadabridge@scadabridge-site-b-b:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/site-b-node-b/appsettings.Site.json
+21 -7
View File
@@ -11,14 +11,21 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-b-a:8082", "akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082" "akka.tcp://scadabridge@scadabridge-site-b-a:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/site-c-node-a/appsettings.Site.json
+19 -5
View File
@@ -14,11 +14,18 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082", "akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082" "akka.tcp://scadabridge@scadabridge-site-c-b:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docker/site-c-node-b/appsettings.Site.json
+21 -7
View File
@@ -11,14 +11,21 @@
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-c-a:8082", "akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082" "akka.tcp://scadabridge@scadabridge-site-c-a:8082"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1 "MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -41,9 +48,16 @@
"SqliteDbPath": "/app/data/store-and-forward.db" "SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ // DEV-ONLY control-plane preshared key — NOT a real secret. Must be
"akka.tcp://scadabridge@scadabridge-central-a:8081", // IDENTICAL on both nodes of the pair and match the central-side entry in
"akka.tcp://scadabridge@scadabridge-central-b:8081" // ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
], ],
"DeploymentTimeout": "00:02:00", "DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30", "LifecycleTimeout": "00:00:30",
docs/components/ClusterInfrastructure.md
+96 -61
View File
@@ -1,23 +1,33 @@
# Cluster Infrastructure # Cluster Infrastructure
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, split-brain resolution, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic. The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
## Overview ## Overview
Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project: Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project:
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, split-brain settings), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system. - **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons. - **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons.
This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host. This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.
Both central and site clusters run this same topology: two nodes, one active (cluster leader), one standby, with automatic failover and no manual intervention required for dual-node recovery. Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery.
## Key Concepts ## Key Concepts
### Active/standby via cluster leadership ### One `ActorSystem` name for every cluster
Akka.NET cluster leadership determines which node is "active". The cluster leader is the oldest node in the cluster, as tracked by the keep-oldest split-brain resolver. `ActiveNodeGate` (in the Host) exposes `IsActiveNode` by checking whether `cluster.SelfMember.Status == MemberStatus.Up` and `cluster.State.Leader == cluster.SelfAddress`. Cluster singletons — which run on the oldest `Up` member — automatically migrate to the surviving node on failover. Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system.
### Active/standby is the oldest `Up` member — never the cluster leader
A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`:
- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`).
- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call.
- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`).
Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.
### Configuration contract vs. bootstrap split ### Configuration contract vs. bootstrap split
@@ -33,22 +43,26 @@ Cluster Infrastructure provides the hosting platform; each singleton is owned an
`AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision. `AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits three additional stanzas: `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`). The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`):
```csharp ```csharp
// Abbreviated — see AkkaHostedService.BuildHocon for the full method. // Abbreviated — see AkkaHostedService.BuildHocon for the full method.
public static string BuildHocon( var downingBlock = string.Equals(
NodeOptions nodeOptions, clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
ClusterOptions clusterOptions, ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
IEnumerable<string> roles, auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
TimeSpan transportHeartbeat, : $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
TimeSpan transportFailure) split-brain-resolver {{
{ active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
var seedNodesStr = string.Join(",", stable-after = {DurationHocon(clusterOptions.StableAfter)}
clusterOptions.SeedNodes.Select(QuoteHocon)); keep-oldest {{
var rolesStr = string.Join(",", roles.Select(QuoteHocon)); down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@" return $@"
audit-telemetry-dispatcher {{ audit-telemetry-dispatcher {{
type = ForkJoinDispatcher type = ForkJoinDispatcher
throughput = 100 throughput = 100
@@ -66,13 +80,7 @@ akka {{
seed-nodes = [{seedNodesStr}] seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}] roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers} min-nr-of-members = {clusterOptions.MinNrOfMembers}
split-brain-resolver {{ {downingBlock}
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
failure-detector {{ failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)} heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)} acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -83,23 +91,35 @@ akka {{
run-by-clr-shutdown-hook = on run-by-clr-shutdown-hook = on
}} }}
}}"; }}";
}
``` ```
A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.
The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors. The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.
### Split-brain resolution Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.
The keep-oldest strategy is the only strategy `ClusterOptionsValidator` permits for ScadaBridge's two-node clusters. Quorum strategies (`keep-majority`, `static-quorum`) cannot distinguish a crash from a partition with two nodes — both sides would be below quorum and both would shut down. Keep-oldest with `down-if-alone = on` ensures at most one node runs the cluster at any time: ### Downing strategy (auto-down — availability-first)
- On a network partition, the older node stays active; the younger node downs itself. **Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
- If the oldest node finds itself alone (no reachable members), it downs itself rather than running in isolation. Without `down-if-alone`, the oldest node could run as a single-node cluster while the younger node forms its own — producing two live clusters with divergent singleton state.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.
`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node.
### Downed-node recovery
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
### Failure detection and failover timeline ### Failure detection and failover timeline
Detection uses two independent Akka heartbeat channels: Detection uses two independent Akka heartbeat channels:
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers `Unreachable` events that the split-brain resolver acts on. - **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on.
- **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`. - **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`.
With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds: With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds:
@@ -107,28 +127,33 @@ With the defaults in `ClusterOptions`, the total failover budget is approximatel
| Phase | Duration | Source | | Phase | Duration | Source |
|-------|----------|--------| |-------|----------|--------|
| Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` | | Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` |
| Split-brain stable-after | 15 s | `ClusterOptions.StableAfter` | | Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` |
| Singleton restart | < 1 s | Actor `PreStart` | | Singleton restart | < 1 s | Actor `PreStart` |
The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.
### Graceful shutdown and singleton handover ### Graceful shutdown and singleton handover
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. `SiteCallAuditActor` has an explicit graceful-stop task registered on `PhaseClusterLeave` with a 10-second timeout to drain any in-flight EF Core upsert before handover opens: When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.
Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:
```csharp ```csharp
siteCallAuditShutdown.AddTask( // SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
Akka.Actor.CoordinatedShutdown.PhaseClusterLeave, Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
"drain-site-call-audit-singleton", $"drain-{name}-singleton",
async () => async () =>
{ {
try try
{ {
await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10)); await manager.GracefulStop(timeout);
} }
catch (Exception ex) catch (Exception ex)
{ {
_logger.LogWarning(ex, logger.LogWarning(ex,
"SiteCallAudit singleton did not drain within the graceful-stop " "{Singleton} singleton did not drain within the graceful-stop timeout; "
+ "timeout; falling through to PoisonPill handover"); + "falling through to PoisonPill handover", name);
} }
return Akka.Done.Instance; return Akka.Done.Instance;
}); });
@@ -136,25 +161,29 @@ siteCallAuditShutdown.AddTask(
### Cluster roles and singleton scoping ### Cluster roles and singleton scoping
Each node carries one or more cluster roles set in the HOCON `roles` list. Site nodes carry both a base `"Site"` role and a site-specific role (`"site-{SiteId}"`, e.g. `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster, not on any other site's nodes. Central singletons use no role scope — all central nodes share the `"Central"` role. Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role.
### Dual-node recovery ### Dual-node recovery
Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. No startup ordering dependency exists, and no manual intervention is required. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with. Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.
### Cluster singletons hosted ### Cluster singletons hosted
The Host wires the following singletons. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern; each singleton's behaviour is documented in the owning component. The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component.
**Central singletons (active central node, no role scope):** **Central singletons (oldest `Up` central node, no role scope):**
| Singleton name | Actor class | Owner | | Singleton name | Actor class | Owner |
|----------------|-------------|-------| |----------------|-------------|-------|
| `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) | | `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) |
| `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) | | `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) |
| `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) | | `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) |
| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) |
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) |
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History |
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) |
**Site singletons (active site node, scoped to `"site-{SiteId}"` role):** **Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):**
| Singleton name | Actor class | Owner | | Singleton name | Actor class | Owner |
|----------------|-------------|-------| |----------------|-------------|-------|
@@ -173,29 +202,29 @@ Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator
services.AddClusterInfrastructure(); services.AddClusterInfrastructure();
``` ```
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section (wrong strategy, `MinNrOfMembers != 1`, `DownIfAlone = false`, fewer than two seed nodes) throws an `OptionsValidationException` at startup rather than booting into a broken cluster. This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there).
### Checking active-node status ### Checking active-node status
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root): Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership:
```csharp ```csharp
// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode public bool IsActiveNode
{ {
get get
{ {
var system = _akkaService.ActorSystem; var system = _akkaService.ActorSystem;
if (system == null) return false; if (system == null)
return false;
var cluster = Cluster.Get(system); var cluster = Cluster.Get(system);
var self = cluster.SelfMember; return ClusterActivityEvaluator.SelfIsOldest(cluster);
if (self.Status != MemberStatus.Up) return false;
var leader = cluster.State.Leader;
return leader != null && leader == self.Address;
} }
} }
``` ```
This returns `false` while the actor system is warming up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes. This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.
## Configuration ## Configuration
@@ -205,13 +234,14 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
| Key | Type | Default | Description | | Key | Type | Default | Description |
|-----|------|---------|-------------| |-----|------|---------|-------------|
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries; both nodes list both themselves and their partner. | | `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. |
| `SplitBrainResolverStrategy` | `string` | `"keep-oldest"` | Must be `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. | | `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Cluster must be stable for this duration before the resolver acts to down unreachable nodes. | | `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. |
| `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. | | `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. |
| `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. | | `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. |
| `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. | | `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. |
| `DownIfAlone` | `bool` | `true` | Must be `true`. See split-brain resolution above. | | `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. |
| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. |
### `ScadaBridge:Node` ### `ScadaBridge:Node`
@@ -241,7 +271,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
"akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081" "akka.tcp://scadabridge@scadabridge-central-b:8081"
], ],
"SplitBrainResolverStrategy": "keep-oldest", "SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15", "StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02", "HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10", "FailureDetectionThreshold": "00:00:10",
@@ -251,7 +281,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
} }
``` ```
`DownIfAlone` is not present in the docker files because its default value of `true` is correct and `ClusterOptionsValidator` rejects `false`. `DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert.
## Dependencies & Interactions ## Dependencies & Interactions
@@ -261,22 +291,26 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
- [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee. - [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component. - [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
- [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`. - [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the cluster leader. - [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member.
- Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md). - Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md).
## Troubleshooting ## Troubleshooting
### Node fails to join cluster on startup ### Node fails to join cluster on startup
`ClusterOptionsValidator` rejects fewer than two seed nodes, a non-`keep-oldest` strategy, `MinNrOfMembers != 1`, or `DownIfAlone = false` at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`. `ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).
### Singleton not starting after failover ### Singleton not starting after failover
If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this. If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this.
### Two live clusters (split-brain) ### Two live clusters (dual-active)
If `DownIfAlone = false` were accepted (the validator rejects it), the oldest node could run alone while the younger forms its own cluster, producing two live clusters with divergent singleton state and dual MS SQL writers on central. `ClusterOptionsValidator` makes this configuration impossible to boot. Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby.
Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage.
### Graceful shutdown takes longer than expected ### Graceful shutdown takes longer than expected
@@ -285,6 +319,7 @@ If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShut
## Related Documentation ## Related Documentation
- [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md) - [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md)
- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md)
- [Host](./Host.md) - [Host](./Host.md)
- [Site Runtime](./SiteRuntime.md) - [Site Runtime](./SiteRuntime.md)
- [Health Monitoring](./HealthMonitoring.md) - [Health Monitoring](./HealthMonitoring.md)
docs/components/Communication.md
+70 -31
View File
@@ -1,6 +1,6 @@
# Central–Site Communication # Central–Site Communication
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides two independent transports — Akka.NET `ClusterClient` for command/control and gRPC server-streaming for real-time data — wired together through a pair of actors that each cluster registers with the `ClusterClientReceptionist`. The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports, **all now gRPC-based across the boundary** (Akka `ClusterClient` was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23): gRPC command/control in both directions (`CentralControlService` on central via `GrpcCentralTransport`; `SiteCommandService` on the site via `GrpcSiteTransport`), gRPC server-streaming for real-time data (`SiteStreamService`), and plain token-gated HTTP for the deployment-config fetch. Cross-cluster Akka remoting and `ClusterClientReceptionist` are gone; Akka remoting is now intra-cluster only.
## Overview ## Overview
@@ -18,27 +18,52 @@ DI registration is called from the Host composition root via `AddCommunication`.
## Key Concepts ## Key Concepts
### Two transports, two concerns ### Three transports, three concerns
| Transport | Direction | Purpose | | Transport | Who dials | Data direction | Purpose |
|-----------|-----------|---------| |-----------|-----------|----------------|---------|
| Akka.NET `ClusterClient` | bidirectional (command/control) | Deployments, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications | | gRPC command/control (`CentralControlService`) | **site dials central** | site → central | Heartbeats, health reports, notification submit/status, audit + cached-telemetry ingest, reconcile |
| gRPC server-streaming (`SiteStreamService`) | site → central | Real-time attribute value and alarm state changes | | gRPC command/control (`SiteCommandService`) | **central dials the site** | central → site | Deploy notifies, lifecycle, OPC UA, remote queries, subscribe/unsubscribe handshake, snapshots, parked, route, failover |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa. The transports are independent. A gRPC stream interruption does not affect in-flight command/control calls, and vice versa.
**`SiteStreamService`'s gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the `SiteStreamService` server** and **central is the client**. Command/control is different: it runs on its own two services — `CentralControlService` (central-hosted, site dials in) and `SiteCommandService` (site-hosted, central dials in). So a central node now DOES host a gRPC server (`CentralControlService`), unlike before Phase 4. The two legacy `Ingest*` unary RPCs on the site-hosted `SiteStreamService` remain dead in the shipped topology (no site dials a site for ingest); audit telemetry is pushed site → central over `CentralControlService`, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site's `SiteStreamService`.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting — unauthenticated, but intra-cluster only.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting is plaintext and open to anything that can reach the remoting port — but as of Phase 4 remoting no longer crosses the site↔central boundary, so this exposure is pair-internal.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listeners are still **h2c** — `ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates the command/control and streaming services alike: every method under `/sitestream.SiteStreamService/` (including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows) **and** the Phase-4 `CentralControlService` (site→central) / `SiteCommandService` (central→site). It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. The caller attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the command message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`), carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
var url = $"{centralFetchBaseUrl.TrimEnd('/')}/api/internal/deployments/{Uri.EscapeDataString(deploymentId)}/config";
using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This path was originally introduced because a flattened config could exceed the default 128 KB Akka frame size when command/control rode ClusterClient — the oversized message was silently dropped and the deploy hung to its timeout (see `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). That Akka frame limit no longer applies now that command/control rides gRPC (`SiteCommandService`, 4 MB cap), but notify-and-fetch remains the deploy path. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline — now over gRPC, so it is no longer at risk of the 128 KB drop.
### Hub-and-spoke topology ### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes. Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `GrpcSiteTransport` channel pair per site (dialling `SiteCommandService`); each site maintains a single `GrpcCentralTransport` channel pair pointed at both central nodes (dialling `CentralControlService`).
### `SiteEnvelope` routing ### `SiteEnvelope` routing
Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site: Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site:
```csharp ```csharp
// CommunicationService.cs — deployment pattern // CommunicationService.cs — deployment pattern (notify-and-fetch)
public async Task<DeploymentStatusResponse> DeployInstanceAsync( public async Task<DeploymentStatusResponse> RefreshDeploymentAsync(
string siteId, DeployInstanceCommand command, CancellationToken cancellationToken = default) string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default)
{ {
var envelope = new SiteEnvelope(siteId, command); var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<DeploymentStatusResponse>( return await GetActor().Ask<DeploymentStatusResponse>(
@@ -70,12 +95,14 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
## Architecture ## Architecture
> **Phase 4 note (2026-07-23).** The `ClusterClient` mechanics described in this section — `SiteEnvelope` routing via `ClusterClient.Send("/user/site-communication", …)`, one `ClusterClient` per site on `CentralCommunicationActor`, the site's outbound `_centralClient`, and `ClusterClientReceptionist` registration — were **replaced by gRPC command/control** in Phase 4 of the ClusterClient→gRPC migration. Central→site commands now go over `GrpcSiteTransport` → the site-hosted `SiteCommandService` (per-site NodeA→NodeB failover channel pair), and site→central messages over `GrpcCentralTransport` → the central-hosted `CentralControlService` (sticky central-a→central-b pair). Endpoints are dialled directly (no receptionist, no cross-cluster actor discovery). The per-site address load below still runs, but it now feeds a **per-site gRPC-endpoint cache** (from `GrpcNodeAAddress`/`GrpcNodeBAddress`) consumed by a `SitePairChannelProvider`, not a `ClusterClient` contact set. See [Component-Communication.md](../requirements/Component-Communication.md) for the current design.
### Central-side: `CentralCommunicationActor` ### Central-side: `CentralCommunicationActor`
`CentralCommunicationActor` is a `ReceiveActor` created at `/user/central-communication` and registered with `ClusterClientReceptionist` so the site's `ClusterClient` can locate it. It owns: `CentralCommunicationActor` is a `ReceiveActor` created at `/user/central-communication` and registered with `ClusterClientReceptionist` so the site's `ClusterClient` can locate it. It owns:
- A `Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)>` keyed by site identifier — one `ClusterClient` per site. - A `Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)>` keyed by site identifier — one `ClusterClient` per site.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database, parses `NodeAAddress` and `NodeBAddress` into Akka receptionist paths (`{addr}/system/receptionist`), and pipes a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` creates, updates, or stops `ClusterClient` actors based on the diff. - A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database and (as of Phase 4) builds a **per-site gRPC-endpoint cache** from `GrpcNodeAAddress`/`GrpcNodeBAddress`, piping a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` reconciles the diff, updating the `SitePairChannelProvider` that `GrpcSiteTransport` uses to dial each site's `SiteCommandService`. (Before Phase 4 this parsed the Akka `NodeAAddress`/`NodeBAddress` into receptionist paths and created/updated/stopped a `ClusterClient` per site.)
- Proxy references to `NotificationOutboxActor` and `AuditLogIngestActor` cluster singletons, injected post-construction via `RegisterNotificationOutbox` / `RegisterAuditIngest` messages from the Host. Messages that arrive before the proxy is registered are answered with a non-accepted ack (notifications) or an empty reply (audit), so the site retries without data loss. - Proxy references to `NotificationOutboxActor` and `AuditLogIngestActor` cluster singletons, injected post-construction via `RegisterNotificationOutbox` / `RegisterAuditIngest` messages from the Host. Messages that arrive before the proxy is registered are answered with a non-accepted ack (notifications) or an empty reply (audit), so the site retries without data loss.
- Fanout of `SiteHealthReport` to the peer central node via `DistributedPubSub`, keyed on the `site-health-replica` topic, so both central nodes' aggregators stay in sync regardless of which central node the site's `ClusterClient` load-balanced the report to. - Fanout of `SiteHealthReport` to the peer central node via `DistributedPubSub`, keyed on the `site-health-replica` topic, so both central nodes' aggregators stay in sync regardless of which central node the site's `ClusterClient` load-balanced the report to.
@@ -86,7 +113,7 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
`SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns: `SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns:
- An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`. - An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`.
- A `Timers`-based heartbeat (default 5-second interval, first tick after 1 second). Each tick sends a `HeartbeatMessage` with `IsActive` stamped from the Akka `Cluster` leader check — the node is active when its `MemberStatus` is `Up` and it holds cluster leadership. - A `Timers`-based heartbeat on `CommunicationOptions.ApplicationHeartbeatInterval` (default 5 s; deliberately distinct from the Akka.Remote `TransportHeartbeatInterval`, so retuning the transport failure detector cannot silently retune the health heartbeat). Each tick sends a `HeartbeatMessage` whose `IsActive` is stamped from `ActiveNodeEvaluator.SelfIsOldestUp` — the node is active when it is the **oldest `Up` member**, *not* when it holds cluster leadership (`SiteCommunicationActor.cs:517-518`). A throwing active-check is caught and reported as `IsActive = false`.
- Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall. - Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall.
Site-to-central messages (health reports, audit batches, notification submissions) are sent via: Site-to-central messages (health reports, audit batches, notification submissions) are sent via:
@@ -108,9 +135,9 @@ A malformed address for one site does not abort the refresh loop — the actor c
### gRPC real-time data transport ### gRPC real-time data transport
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, a gRPC server-streaming service defined in `sitestream.proto`. Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, defined in `sitestream.proto`. The **server runs on every site node and the client runs on central** — central dials in to receive the stream (see the transport table above).
**Site-side** — `SiteStreamGrpcServer` (Kestrel HTTP/2, port 8083): **Site-side** — `SiteStreamGrpcServer` (Kestrel h2c, HTTP/2 only, port 8083):
- Implements `SiteStreamService.SiteStreamServiceBase`. - Implements `SiteStreamService.SiteStreamServiceBase`.
- For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly). - For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly).
@@ -144,7 +171,7 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
**Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`: **Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`:
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per site identifier. On `GetOrCreate`, it compares the cached client's `Endpoint` to the requested endpoint and atomically replaces a stale client (different endpoint — NodeA→NodeB failover flip, or an edited address) with a fresh one. - `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per **`(site, endpoint)` pair** — a `ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient>`. The key was widened from site-only to fix an arch-review High: with a site-only key, one debug session's NodeA→NodeB failover flip disposed a channel another session was still using. `GetOrCreate` therefore no longer disposes on endpoint mismatch; both of a site's node channels coexist, and site *removal* (`RemoveSiteAsync`) is the only shared-disposal path. The trade-off is that an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site.
- `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread. - `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread.
### Debug stream session lifecycle ### Debug stream session lifecycle
@@ -162,18 +189,22 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
### Proto definition summary ### Proto definition summary
```proto ```proto
// Protos/sitestream.proto // Protos/sitestream.proto — six RPCs, all served by the SITE
service SiteStreamService { service SiteStreamService {
rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent); rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent);
rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent);
rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck); rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck);
rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck); rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse); rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse);
rpc PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse);
} }
``` ```
`SubscribeInstance` carries the real-time data stream. The other three RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`) serve the Audit Log component's gRPC telemetry push and reconciliation pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. Two are server-streaming: `SubscribeInstance` carries the per-instance real-time stream; `SubscribeSite` is the **site-wide, alarm-only** stream (no instance filter, attribute updates never carried) that feeds the aggregated central live alarm cache. The four unary RPCs serve the Audit Log and Site Call Audit push/pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. As noted above, the two `Ingest*` RPCs are dead in the shipped topology (no central gRPC server exists for a site to dial); the two `Pull*` RPCs are live, with central as the caller.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–21) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. `SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–23) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. Field numbers are never reused and evolution is additive only.
The generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out, so editing `sitestream.proto` does not regenerate on build — regeneration is a manual toggle-build-copy-untoggle.
## Usage ## Usage
@@ -181,10 +212,10 @@ Central callers interact through `CommunicationService`, which wraps each comman
| Pattern | Method | Timeout | | Pattern | Method | Timeout |
|---------|--------|---------| |---------|--------|---------|
| Instance deployment | `DeployInstanceAsync` | 120 s | | Instance deployment (notify-and-fetch) | `RefreshDeploymentAsync` | 120 s |
| Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s | | Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s |
| Artifact deployment | `DeployArtifactsAsync` | 60 s | | Artifact deployment | `DeployArtifactsAsync` | 60 s |
| Integration routing | `RouteIntegrationCallAsync` | 30 s | | Integration routing (Inbound API routed-site-script) | `RouteToCallAsync`, `RouteToGetAttributesAsync`, `RouteToSetAttributesAsync`, `RouteToWaitForAttributeAsync` | 30 s (`IntegrationTimeout`) |
| Debug snapshot | `RequestDebugSnapshotAsync` | 30 s | | Debug snapshot | `RequestDebugSnapshotAsync` | 30 s |
| Remote queries | `QueryEventLogsAsync`, `QueryParkedMessagesAsync`, etc. | 30 s | | Remote queries | `QueryEventLogsAsync`, `QueryParkedMessagesAsync`, etc. | 30 s |
| OPC UA tag browse | `BrowseNodeAsync` | 30 s | | OPC UA tag browse | `BrowseNodeAsync` | 30 s |
@@ -197,24 +228,28 @@ For real-time streaming, callers use `DebugStreamService.StartStreamAsync`, whic
## Configuration ## Configuration
All options are bound from the `Communication` section via `CommunicationOptions`: All options are bound from the `ScadaBridge:Communication` section via `CommunicationOptions`:
| Key | Default | Description | | Key | Default | Description |
|-----|---------|-------------| |-----|---------|-------------|
| `DeploymentTimeout` | `00:02:00` | Ask timeout for instance deployment commands. | | `DeploymentTimeout` | `00:02:00` | Ask timeout for the `RefreshDeploymentCommand` round-trip (covers the site's HTTP config fetch and apply). |
| `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). | | `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). |
| `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. | | `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. |
| `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. | | `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. |
| `IntegrationTimeout` | `00:00:30` | Ask timeout for integration routing and Inbound API routing. | | `IntegrationTimeout` | `00:00:30` | Ask timeout for integration routing and Inbound API routing. |
| `DebugViewTimeout` | `00:00:10` | Ask timeout for debug subscribe/unsubscribe handshake. | | `DebugViewTimeout` | `00:00:10` | Ask timeout for debug subscribe/unsubscribe handshake. |
| `NotificationForwardTimeout` | `00:00:30` | Ask timeout for notification submission forwarding. | | `NotificationForwardTimeout` | `00:00:30` | Ask timeout for notification submission forwarding. |
| `CentralContactPoints` | `[]` | Site-side: Akka addresses of central nodes, e.g. `akka.tcp://scadabridge@central-a:8081`. | | `CentralGrpcEndpoints` | `[]` | Site-side: gRPC h2c endpoints of the central nodes' `CentralControlService`, e.g. `http://scadabridge-central-a:8083` (central's `CentralGrpcPort`, default 8083 — direct, not via Traefik, which is HTTP/1 only). **Required on Site nodes** (at least one); empty on Central nodes, which host `CentralControlService` and do not dial it. Replaces the former `CentralContactPoints` Akka-address list. |
| `GrpcKeepAlivePingDelay` | `00:00:15` | HTTP/2 keepalive PING interval on `SiteStreamGrpcClient`. | | `GrpcKeepAlivePingDelay` | `00:00:15` | HTTP/2 keepalive PING interval on `SiteStreamGrpcClient`. |
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. | | `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. | | `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
| `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. | | `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. |
| `TransportHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` heartbeat cadence. | | `ApplicationHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` site→central heartbeat cadence. |
| `TransportHeartbeatInterval` | `00:00:05` | Akka.Remote transport failure-detector heartbeat interval (emitted into the HOCON by the Host). Distinct from the application heartbeat above. |
| `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. | | `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. |
| `CentralFetchBaseUrl` | `""` | Base URL (Traefik/LB) the site uses to fetch deploy configs from central. Carried in `RefreshDeploymentCommand` so sites need no standing config; **empty makes a deploy impossible** — `DeploymentService` fails fast. |
| `PendingDeploymentTtl` | `00:05:00` | How long a staged `PendingDeployment` row and its fetch token stay valid. Must comfortably cover both site nodes' fetches within one deploy window. |
| `PendingDeploymentPurgeInterval` | `01:00:00` | Cadence of the central `pending-deployment-purge` singleton that sweeps TTL-expired staging rows. Hygiene only — the fetch endpoint already enforces the TTL. |
Three layers of dead-client detection protect the gRPC stream path: Three layers of dead-client detection protect the gRPC stream path:
@@ -227,27 +262,27 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions ## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project. - [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the active/standby leader model that `SiteCommunicationActor`'s `IsActive` check and `CentralCommunicationActor`'s `DistributedPubSub` fanout both depend on. - [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides ClusterSingleton and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name for intra-cluster remoting. (`ClusterClientReceptionist` is no longer used — cross-cluster messaging is gRPC as of Phase 4.) `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`. - [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deployments, lifecycle commands, and artifact deployments to sites. - [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly. - [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync. - [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts `IngestAuditEvents`, `IngestCachedTelemetry`, and `PullAuditEvents` RPCs. `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` ClusterClient messages to the `AuditLogIngestActor` proxy. - [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. The `Ingest*` pair on the site-hosted service stays unused in the shipped topology; sites push audit telemetry site→central over gRPC to `CentralControlService`, which routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site's `SiteStreamService`.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls. - [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`. - [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way. - [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. - [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not any cluster transport. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) After Phase 4 this component uses no `ClusterClient` at all — its cross-cluster connections are the gRPC `CentralControlService` / `SiteCommandService` transports. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the HTTP deploy-config transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md). - Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting ## Troubleshooting
### A site's commands fail immediately ### A site's commands fail immediately
Check that `NodeAAddress` and `NodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `ClusterClient` is created and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added. Check that `GrpcNodeAAddress` and `GrpcNodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `SiteCommandService` channel is built and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
### Commands are timing out but the site is reachable ### Commands are timing out but the site is reachable
A single malformed address string for one site can silently prevent `ClusterClient` creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The actor parse-guard catches the `ActorPath.Parse` exception per-site so the rest of the refresh proceeds. A single malformed gRPC endpoint string for one site can silently prevent channel creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The per-site parse-guard skips the bad entry so the rest of the refresh proceeds.
A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means `LoadSiteAddressesFromDb` itself threw (typically a SQL connection error); the cache is left stale until the next successful refresh. A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means `LoadSiteAddressesFromDb` itself threw (typically a SQL connection error); the cache is left stale until the next successful refresh.
@@ -257,6 +292,10 @@ A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means
After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session. After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session.
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`) but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not ### Heartbeats arrive but health reports do not
`SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`. `SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`.
docs/components/DeploymentManager.md
+20 -6
View File
@@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers:
- **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call. - **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call.
- **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment. - **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment.
The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running. The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip. Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip.
### Notify-and-fetch: the config does not travel in the Akka message
A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path.
This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit.
Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour).
The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change.
### Per-instance operation lock ### Per-instance operation lock
`OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel. `OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel.
@@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment
3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written. 3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written.
4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent. 4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent.
5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI. 5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI.
6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`. 6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before.
7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails. 7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails.
8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`. 8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`.
9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user. 9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user.
@@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 6–7) writes `DeploymentStatus.Fail
```csharp ```csharp
// DeploymentService.DeployInstanceAsync — exception handler // DeploymentService.DeployInstanceAsync — exception handler
var isTimeout = ex is TimeoutException or OperationCanceledException; var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException;
record.Status = DeploymentStatus.Failed; record.Status = DeploymentStatus.Failed;
record.ErrorMessage = isTimeout record.ErrorMessage = isTimeout
@@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De
- [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view. - [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows. - [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. - [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface. - [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active. - [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active.
- [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR. - [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. - [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s.
- [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`. - [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`.
## Troubleshooting ## Troubleshooting
@@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent
The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one. The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one.
### A deployment fails with a config-fetch error
The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout.
### DeleteOrphaned audit entry ### DeleteOrphaned audit entry
The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry. The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.
docs/components/ManagementService.md
+7 -4
View File
@@ -56,17 +56,18 @@ Mutating handlers that call repositories directly invoke `AuditAsync` (backed by
### Actor lifecycle and registration ### Actor lifecycle and registration
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and registers it with `ClusterClientReceptionist`: `AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and publishes it to `ManagementActorHolder`, which is the only way anything reaches it:
```csharp ```csharp
var mgmtActor = _actorSystem!.ActorOf( var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)), Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)),
"management"); "management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>(); var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor; mgmtHolder.ActorRef = mgmtActor;
``` ```
A `ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor)` call sat between those two statements until 2026-07-22. It was deleted because nothing ever sent to it: the CLI it was built for uses HTTP, not ClusterClient.
`ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world. `ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world.
The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later. The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later.
@@ -154,9 +155,11 @@ Content-Type: application/json
A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`. A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`.
### Sending a command via ClusterClient ### Sending a command in-process
The `ManagementActor` is also reachable from any `ClusterClient` that has a contact point into the central cluster. The actor is registered under `/system/receptionist` with the path `/user/management`. Callers construct and `Tell` a `ManagementEnvelope` and expect one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEnvelope` is also the in-process contract: a caller holding `ManagementActorHolder.ActorRef` asks the actor directly and expects one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEndpoints` is that caller.
There is **no** out-of-process actor path. The actor was advertised via `ClusterClientReceptionist` until 2026-07-22, so a `ClusterClient` with a contact point into the central cluster could `Tell` it a `ManagementEnvelope`; no caller ever did, and the registration is gone. The HTTP endpoints above are the only remote management surface.
## Command Groups ## Command Groups
docs/components/TreeView.md
+23 -1
View File
@@ -41,9 +41,31 @@ In `Single` mode the component uses `SelectedKey` / `SelectedKeyChanged` (two-wa
When `ContextMenu` is non-null, right-clicking any row suppresses the browser default and positions a Bootstrap `dropdown-menu show` div at the cursor coordinates using `position: fixed`. An invisible overlay behind the menu dismisses it on click-outside; Escape also dismisses it. The menu receives the `TItem` of the right-clicked node, so the consumer's fragment can branch on node type. When `ContextMenu` is non-null, right-clicking any row suppresses the browser default and positions a Bootstrap `dropdown-menu show` div at the cursor coordinates using `position: fixed`. An invisible overlay behind the menu dismisses it on click-outside; Escape also dismisses it. The menu receives the `TItem` of the right-clicked node, so the consumer's fragment can branch on node type.
### Keyboard navigation & accessibility (WAI-ARIA tree pattern)
Delivered 2026-08-01 (M10 residual R7). The component implements the full [WAI-ARIA tree pattern](https://www.w3.org/WAI/ARIA/apg/patterns/treeview/).
**Roving tabindex.** Exactly one `li[role="treeitem"]` carries `tabindex="0"` at a time; every other node is `tabindex="-1"`, so the whole tree is a single Tab stop. The target is resolved once per render by `ResolveTabbableKey()`: the node the user last landed on (`_focusedKey`) while it is still visible → else the currently `SelectedKey` node → else the first visible node. A keyboard move sets `_focusNeedsApply`, and `OnAfterRenderAsync` pulls browser focus onto the new node via a per-node `ElementReference`, guarded by the same `JSException` / `JSDisconnectedException` / `InvalidOperationException` triple the context-menu focus uses (under bUnit there is no real focus, so it is a safe no-op).
| Key | Behaviour |
| --- | --- |
| `ArrowDown` / `ArrowUp` | Move to the next / previous **visible** node (a collapsed branch's children are skipped). No-op at the ends. |
| `ArrowRight` | Collapsed branch → expand (focus stays); expanded branch → move to first child; leaf → no-op. |
| `ArrowLeft` | Expanded branch → collapse (focus stays); otherwise → move to parent (via `BuildParentLookup()`). No-op at a root leaf. |
| `Home` / `End` | Move to the first / last visible node. |
| `Enter` / `Space` | Activate the node — routed to the *same* path a mouse click takes (`OnContentClick` in `Single` mode, `OnCheckboxToggle` in `Checkbox` mode), so selection semantics never diverge between input modes. |
The visible order is produced by `BuildVisibleNodes()`, which mirrors `RenderNode`'s visibility rules exactly — **the two must stay in step**, or arrow navigation will skip or invent rows.
**ARIA attributes.** `ul[role=tree]` root, `ul[role=group]` for children, `li[role=treeitem]` nodes carrying `aria-level` (1-based), `aria-posinset`, `aria-setsize`, `aria-expanded` (branches only), and `aria-selected` (selectable / checkbox trees only — absent on a non-selectable tree).
**Event scoping.** `@onkeydown:stopPropagation` sits on the `<li>` (so a nested node's keypress is not also handled by its ancestors) and on the chevron, the checkbox, and the `.tv-content` slot — so a consumer's own buttons and inputs inside `NodeContent` keep their key handling. Browser scroll-on-Space/Arrow is suppressed by a small **native** inline `onkeydown` on the root `<ul>` that calls `preventDefault()` only for the navigation keys and only when the event target is the treeitem itself; Blazor's `preventDefault` directive is all-or-nothing per element, so putting it on the `<li>` would both trap Tab inside the tree and cancel Enter/Space activation of consumer buttons. (The app sets no Content-Security-Policy, so the inline handler executes.)
Covered by `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Shared/TreeViewKeyboardNavigationTests.cs` (31 bUnit tests).
## Architecture ## Architecture
The component is a single `@typeparam` `.razor` file with a private `void RenderNode(TItem item, int depth)` local function that recurses the tree at render time — no intermediate view model is built inside the component. Every `<li>` carries `@key="key"` so Blazor can diff the list efficiently. The component is a single `@typeparam` `.razor` file with a private `void RenderNode(TItem item, int depth, int posInSet, int setSize)` local function that recurses the tree at render time — no intermediate view model is built inside the component. Every `<li>` carries `@key="key"` so Blazor can diff the list efficiently.
`IJSRuntime` is injected for two purposes: reading/writing `sessionStorage` for expansion persistence, and setting `input.indeterminate` for tri-state checkboxes. Both call sites guard `JSDisconnectedException` so a disconnected circuit never throws out of the lifecycle methods. `IJSRuntime` is injected for two purposes: reading/writing `sessionStorage` for expansion persistence, and setting `input.indeterminate` for tri-state checkboxes. Both call sites guard `JSDisconnectedException` so a disconnected circuit never throws out of the lifecycle methods.
docs/deployment/production-checklist.md
+4 -1
View File
@@ -44,6 +44,8 @@
- [ ] Windows Service account has minimum necessary permissions - [ ] Windows Service account has minimum necessary permissions
- [ ] Log directory permissions restrict access to service account and administrators - [ ] Log directory permissions restrict access to service account and administrators
- [ ] SMTP credentials use OAuth2 Client Credentials (preferred) or secure Basic Auth - [ ] SMTP credentials use OAuth2 Client Credentials (preferred) or secure Basic Auth
- [ ] EWS transport (`Transport=Ews`): endpoint is an absolute `https://` URL and auth mode is Basic — Basic requires TLS and this is enforced at the write gate, the delivery adapter, and the sender
- [ ] EWS transport: the Exchange service-account password is rotated on the account-owner's schedule, and the SMTP configuration row is updated in the same change
- [ ] API keys for Inbound API are generated with sufficient entropy (32+ chars) - [ ] API keys for Inbound API are generated with sufficient entropy (32+ chars)
### Network ### Network
@@ -51,7 +53,8 @@
- [ ] DNS resolution works between all cluster nodes - [ ] DNS resolution works between all cluster nodes
- [ ] Firewall rules permit Akka.NET remoting (TCP 8081) - [ ] Firewall rules permit Akka.NET remoting (TCP 8081)
- [ ] Firewall rules permit LDAP (TCP 636 for LDAPS) - [ ] Firewall rules permit LDAP (TCP 636 for LDAPS)
- [ ] Firewall rules permit SMTP (TCP 587 for TLS) - [ ] Firewall rules permit SMTP (TCP 587 for TLS) — SMTP transport only
- [ ] EWS transport (`Transport=Ews`): firewall rules permit outbound HTTPS (TCP 443) from central nodes to the Exchange CAS instead of SMTP 587
- [ ] Firewall rules permit SQL Server (TCP 1433) from central nodes only - [ ] Firewall rules permit SQL Server (TCP 1433) from central nodes only
- [ ] Load balancer health check configured against `/health/ready` - [ ] Load balancer health check configured against `/health/ready`
docs/deployment/topology-guide.md
+55 -7
View File
@@ -88,17 +88,19 @@ Both central nodes must be configured as seed nodes for each other:
}, },
"Cluster": { "Cluster": {
"SeedNodes": [ "SeedNodes": [
"akka.tcp://scadabridge@central-01.example.com:8081", "akka.tcp://scadabridge@central-02.example.com:8081",
"akka.tcp://scadabridge@central-02.example.com:8081" "akka.tcp://scadabridge@central-01.example.com:8081"
] ]
} }
} }
} }
``` ```
> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### Cluster Behavior ### Cluster Behavior
- **Split-brain resolver**: Keep-oldest with `down-if-alone = on`, 15-second stable-after. - **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster. - **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
- **Failure detection**: 2-second heartbeat interval, 10-second threshold. - **Failure detection**: 2-second heartbeat interval, 10-second threshold.
- **Total failover time**: ~25 seconds from node failure to singleton migration. - **Total failover time**: ~25 seconds from node failure to singleton migration.
@@ -145,9 +147,11 @@ Each site has its own two-node cluster:
} }
``` ```
> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.
### Site Cluster Behavior ### Site Cluster Behavior
- Same split-brain resolver as central (keep-oldest). - Same split-brain resolver as central (`auto-down`, per the 2026-07-21 decision — see the Central Cluster Behavior note above).
- Singleton actors: Site Deployment Manager migrates on failover. - Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms. - Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by - SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
@@ -174,9 +178,53 @@ database from its peer rather than letting it rejoin.
### Central-Site Communication ### Central-Site Communication
- Sites connect to central via Akka.NET remoting. Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
- The `Communication:CentralSeedNode` setting in the site config points to one of the central nodes. in Phase 4 of the ClusterClient→gRPC migration (2026-07-23), and Akka remoting no longer crosses the
- If that central node is down, the site's communication actor will retry until it connects to the active central node. boundary at all:**
- **gRPC command/control** — both directions, on sticky-failover channel pairs, dialled directly (no
receptionist, no "active central" to identify — each side dials both of the peer's node endpoints):
- *Site → central* to the central-hosted **`CentralControlService`** (`GrpcCentralTransport`): the
site lists the central nodes' gRPC endpoints in `ScadaBridge:Communication:CentralGrpcEndpoints`
(e.g. `http://scadabridge-central-a:8083`, the central's `CentralGrpcPort`, default 8083 — direct
h2c, **not** via Traefik, which is HTTP/1 only). A Site node must list at least one; central nodes
leave it empty.
- *Central → site* to the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`): central dials
the site's `GrpcNodeAAddress` / `GrpcNodeBAddress` (from the Site entity), NodeA→NodeB failover.
- **gRPC streaming + audit pull** — real-time data and audit/telemetry pull on the site-hosted
**`SiteStreamService`**. Note the direction is inverted from the data flow: each **site node hosts
the server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.
| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |
The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.
One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.
**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.
The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.
## Scaling Guidelines ## Scaling Guidelines
docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md
+19
View File
@@ -31,6 +31,25 @@ Any deployment replicating wide rows must size that key deliberately; see the Ph
Note the failure mode differs from the one documented below: an oversized gRPC message is Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped. **rejected**, not silently dropped.
## Amendment (2026-07-23) — Akka frame-size class RETIRED for site↔central command/control
Phase 4 of the ClusterClient→gRPC migration deleted the Akka `ClusterClient` site↔central transport
entirely. **All site↔central command/control now rides gRPC** — central→site over `SiteCommandService`
(`GrpcSiteTransport`) and site→central over `CentralControlService` (`GrpcCentralTransport`), each with
the gRPC default 4 MB message cap and **no per-message Akka frame limit**. Consequently:
- The 128 KB Akka `maximum-frame-size` constraint **no longer applies to any site↔central command
message**, including `DeployArtifactsCommand`, which still carries its payload inline but now travels
over gRPC.
- The **silent frame-drop failure mode** described below — the transport dropping one oversized message
while heartbeats keep flowing and the deploy hangs to its timeout — **cannot occur on that path any
more.** An over-cap gRPC message is *rejected* with an error, not silently discarded.
- The notify-and-fetch deploy path (the 2026-06-26 resolution) still stands and remains the deploy
mechanism; it is simply no longer the *only* thing keeping large payloads off a frame-limited hop.
- Akka remoting is now intra-cluster only, so its frame size governs only pair-internal traffic.
See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about. The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary ## Summary
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md
+68
View File
@@ -0,0 +1,68 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** RESOLVED (DELETED 2026-07-23) · **Tracked:** Gitea
[#32](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/32) (filed 2026-07-23) · **Severity:**
Low (no runtime impact — the path cannot be reached) · **Area:** Central–Site Communication
> **Resolution (2026-07-23):** Decision = **delete** (option 1 below). Removed the
> `IntegrationCallRequest`/`IntegrationCallResponse` messages, `CommunicationService.RouteIntegrationCallAsync`,
> the `SiteCommunicationActor` receive block + `_integrationHandler` field + `LocalHandlerType.Integration`,
> and the four tests that covered them. `IntegrationTimeout` was **kept** — it is the live timeout for the
> Inbound API's `RouteTo*` verbs, which are the actual implementation of this "External → Central → Site →
> Central" pattern (design §4). The narrative comments that documented the exclusion (proto/mapper/dispatcher)
> were updated. Full solution build clean; Communication suite 634 green. This note is retained as the record.
## What
"Pattern 4: Integration Routing" — `CommunicationService.RouteIntegrationCallAsync` →
`SiteEnvelope(IntegrationCallRequest)` → `SiteCommunicationActor` → an integration handler — is
plumbed end to end but connected at neither end.
- **No producer.** `RouteIntegrationCallAsync` (`CommunicationService.cs`, "Pattern 4") has **zero
callers** in `src/` or `tests/`. It is the only one of `CommunicationService`'s command methods
with none.
- **No handler.** `SiteCommunicationActor` forwards to `_integrationHandler` when one is
registered, but `RegisterLocalHandler(LocalHandlerType.Integration, …)` appears **only** in
`SiteCommunicationActorTests.cs`. `AkkaHostedService` registers the other three handler types
(`Artifacts`, `EventLog`, `ParkedMessages`) and never this one.
So if anything ever did call it, the site would answer
`IntegrationCallResponse(Success: false, Error: "Integration handler not available")`
(`SiteCommunicationActor.cs`, Pattern 4) — and the two tests that exercise the path both register
the handler themselves first, which is why the suite has never noticed.
Do not confuse this with the **Inbound API**'s routed-site-script path, which is live, tested, and
uses different messages entirely. This is a separate, unused routing pattern that predates it.
## Why it is recorded rather than fixed
Found during the recon for the ClusterClient→gRPC transport migration
([`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`](../plans/2026-07-22-clusterclient-to-grpc-plan.md),
T0.2), which had to enumerate every command crossing the site↔central boundary. Of the **29**
command types, this is the one that is excluded: **28 migrate to the gRPC contract.**
Porting it would mean designing a proto contract, a `oneof` slot and round-trip mapper tests for a
verb no caller can invoke and no site can service — and enshrining it on a wire format whose
evolution rules are additive-only, so an unused RPC slot is permanent. Deleting it during a
transport migration would mix an unrelated behavioural change into a change whose whole value is
that behaviour is identical. Hence: excluded from the contract, behaviour untouched, decision
deferred to its own change.
## Decision needed
Either:
1. **Delete** — remove `RouteIntegrationCallAsync`, the `IntegrationCallRequest`/`Response`
messages, the `SiteCommunicationActor` receive block, `LocalHandlerType.Integration`, and the
three tests that cover them. This is the default if no consumer is planned.
2. **Wire** — register a real integration handler on site nodes and give the method a caller. This
only makes sense if there is a requirement it serves; none is recorded in
`docs/requirements/`.
Whichever is chosen, do it **before Phase 4** of the migration, since Phase 4 deletes the Akka
transport underneath this path. If it is still dead at that point, option 1 is forced.
## Filing
To be filed as a Gitea issue on `dohertj2/scadabridge` by the repo owner — this note is the
in-repo record of the finding and of the migration exclusion it justifies.
docs/operations/2026-07-16-secrets-clustered-master-key.md
+12
View File
@@ -1,5 +1,17 @@
# Secrets: Clustered Master-Key Posture (Central Pair) # Secrets: Clustered Master-Key Posture (Central Pair)
> **Update 2026-08-07 (truth sweep):** two claims below are stale.
> (1) "there is no built-in cross-node replication today" — cross-node replication
> has since shipped: ScadaBridge adopted opt-in **SQL-Server hub replication** for
> the host secret store (commit `8e12f994`, "feat(secrets): opt-in SQL-Server hub
> replication for the host secret store"; `ZB.MOM.WW.Secrets` 0.2.x Replicator
> packages), covered by the shared library's clustered-secrets runbook
> (`scadaproj/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`).
> (2) the G-8 KEK-rotation runbook is no longer "not yet built" — it ships with the
> shared library at `scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`
> (lib 0.1.3, `Rewrap`/`rewrap-all`). The interim shared-volume posture below
> remains valid but is no longer the only option.
## Purpose ## Purpose
`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a `ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
docs/plans/2026-03-16-cluster-infrastructure-refinement-design.md
+1 -1
View File
@@ -2,7 +2,7 @@
**Date**: 2026-03-16 **Date**: 2026-03-16
**Component**: Cluster Infrastructure (`docs/requirements/Component-ClusterInfrastructure.md`) **Component**: Cluster Infrastructure (`docs/requirements/Component-ClusterInfrastructure.md`)
**Status**: Approved **Status**: Approved — superseded in part: the keep-oldest SBR decision was replaced by the auto-down decision 2026-07-21 (`docs/plans/2026-07-21-auto-down-availability-decision.md`).
## Problem ## Problem
docs/plans/2026-03-16-communication-layer-refinement-design.md
+1 -1
View File
@@ -2,7 +2,7 @@
**Date**: 2026-03-16 **Date**: 2026-03-16
**Component**: Central–Site Communication (`docs/requirements/Component-Communication.md`) **Component**: Central–Site Communication (`docs/requirements/Component-Communication.md`)
**Status**: Approved **Status**: Approved — transport decisions superseded by the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`); the "no buffering at central" decision still stands.
## Problem ## Problem
docs/plans/2026-03-17-deploy-artifacts-remove-configdb.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{ {
"planPath": "docs/plans/2026-03-17-deploy-artifacts-remove-configdb.md", "planPath": "docs/plans/2026-03-17-deploy-artifacts-remove-configdb.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [ "tasks": [
{"id": 4, "subject": "Task 1: Create SiteExternalSystemRepository", "status": "pending"}, {"id": 4, "subject": "Task 1: Create SiteExternalSystemRepository", "status": "completed"},
{"id": 5, "subject": "Task 2: Create SiteNotificationRepository", "status": "pending"}, {"id": 5, "subject": "Task 2: Create SiteNotificationRepository", "status": "completed", "note": "Shipped as planned, then deliberately removed 2026-07-10 (arch-review 08 §1.3 — excise vestigial site notification surface; notification delivery is central-only). The notification_lists / smtp_configurations tables survive on the site as deliberately-empty, unregistered (non-replicated) tables."},
{"id": 6, "subject": "Task 3: Add data connections to DeployArtifactsCommand", "status": "pending"}, {"id": 6, "subject": "Task 3: Add data connections to DeployArtifactsCommand", "status": "completed"},
{"id": 7, "subject": "Task 4: Wire site-local repositories into DI", "status": "pending", "blockedBy": [4, 5, 6]}, {"id": 7, "subject": "Task 4: Wire site-local repositories into DI", "status": "completed", "blockedBy": [4, 5, 6]},
{"id": 8, "subject": "Task 5: Add Deploy Artifacts button to Sites admin page", "status": "pending", "blockedBy": [9]}, {"id": 8, "subject": "Task 5: Add Deploy Artifacts button to Sites admin page", "status": "completed", "blockedBy": [9]},
{"id": 9, "subject": "Task 6: Update ArtifactDeploymentService to include all artifact types", "status": "pending"}, {"id": 9, "subject": "Task 6: Update ArtifactDeploymentService to include all artifact types", "status": "completed"},
{"id": 10, "subject": "Task 7: End-to-end test", "status": "pending", "blockedBy": [7, 8, 9]} {"id": 10, "subject": "Task 7: End-to-end test", "status": "completed", "blockedBy": [7, 8, 9]}
], ],
"lastUpdated": "2026-03-17T16:40:00Z" "lastUpdated": "2026-08-01T00:00:00Z"
} }
docs/plans/2026-03-17-management-service-cli.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{ {
"planPath": "docs/plans/2026-03-17-management-service-cli.md", "planPath": "docs/plans/2026-03-17-management-service-cli.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [ "tasks": [
{"id": 11, "subject": "Task 1: Create ManagementService project and test project", "status": "pending"}, {"id": 11, "subject": "Task 1: Create ManagementService project and test project", "status": "completed"},
{"id": 12, "subject": "Task 2: Define management message contracts in Commons", "status": "pending"}, {"id": 12, "subject": "Task 2: Define management message contracts in Commons", "status": "completed"},
{"id": 13, "subject": "Task 3: Implement ManagementActor", "status": "pending", "blockedBy": [11, 12]}, {"id": 13, "subject": "Task 3: Implement ManagementActor", "status": "completed", "blockedBy": [11, 12]},
{"id": 14, "subject": "Task 4: Register ManagementActor on Central with ClusterClientReceptionist", "status": "pending", "blockedBy": [13]}, {"id": 14, "subject": "Task 4: Register ManagementActor on Central with ClusterClientReceptionist", "status": "completed", "blockedBy": [13], "note": "OBSOLETE — never built; CLI shipped HTTP-only (ManagementHttpClient), receptionist deleted in the 2026-07-22 gRPC migration. Marked completed only to close the tracker."},
{"id": 15, "subject": "Task 5: Create CLI project with ClusterClient scaffolding", "status": "pending", "blockedBy": [12]}, {"id": 15, "subject": "Task 5: Create CLI project with ClusterClient scaffolding", "status": "completed", "blockedBy": [12], "note": "OBSOLETE — never built; CLI shipped HTTP-only (ManagementHttpClient), receptionist deleted in the 2026-07-22 gRPC migration. The CLI project itself exists; only the ClusterClient scaffolding is obsolete."},
{"id": 16, "subject": "Task 6: Implement CLI command groups", "status": "pending", "blockedBy": [15]}, {"id": 16, "subject": "Task 6: Implement CLI command groups", "status": "completed", "blockedBy": [15]},
{"id": 17, "subject": "Task 7: Write ManagementActor unit tests", "status": "pending", "blockedBy": [13]}, {"id": 17, "subject": "Task 7: Write ManagementActor unit tests", "status": "completed", "blockedBy": [13]},
{"id": 18, "subject": "Task 8: End-to-end integration test", "status": "pending", "blockedBy": [14, 16, 17]} {"id": 18, "subject": "Task 8: End-to-end integration test", "status": "completed", "blockedBy": [14, 16, 17]}
], ],
"lastUpdated": "2026-03-17T17:00:00Z" "lastUpdated": "2026-08-01T00:00:00Z"
} }
docs/plans/2026-03-21-grpc-streaming-channel.md.tasks.json
+14 -13
View File
@@ -1,18 +1,19 @@
{ {
"planPath": "docs/plans/2026-03-21-grpc-streaming-channel.md", "planPath": "docs/plans/2026-03-21-grpc-streaming-channel.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code — feature shipped and later extended by the full ClusterClient→gRPC migration (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md)",
"tasks": [ "tasks": [
{"id": 0, "taskId": "1", "subject": "Task 0: Proto Definition & Stub Generation", "status": "pending"}, {"id": 0, "taskId": "1", "subject": "Task 0: Proto Definition & Stub Generation", "status": "completed"},
{"id": 1, "taskId": "2", "subject": "Task 1: Site Config — GrpcPort in NodeOptions", "status": "pending", "blockedBy": [0]}, {"id": 1, "taskId": "2", "subject": "Task 1: Site Config — GrpcPort in NodeOptions", "status": "completed", "blockedBy": [0]},
{"id": 2, "taskId": "3", "subject": "Task 2: Site Entity — gRPC Address Fields", "status": "pending", "blockedBy": [0]}, {"id": 2, "taskId": "3", "subject": "Task 2: Site Entity — gRPC Address Fields", "status": "completed", "blockedBy": [0]},
{"id": 3, "taskId": "4", "subject": "Task 3: Site-Side gRPC Server — StreamRelayActor", "status": "pending", "blockedBy": [0]}, {"id": 3, "taskId": "4", "subject": "Task 3: Site-Side gRPC Server — StreamRelayActor", "status": "completed", "blockedBy": [0]},
{"id": 4, "taskId": "5", "subject": "Task 4: Site-Side gRPC Server — SiteStreamGrpcServer", "status": "pending", "blockedBy": [3]}, {"id": 4, "taskId": "5", "subject": "Task 4: Site-Side gRPC Server — SiteStreamGrpcServer", "status": "completed", "blockedBy": [3]},
{"id": 5, "taskId": "6", "subject": "Task 5: Switch Site Host to WebApplicationBuilder + gRPC", "status": "pending", "blockedBy": [4]}, {"id": 5, "taskId": "6", "subject": "Task 5: Switch Site Host to WebApplicationBuilder + gRPC", "status": "completed", "blockedBy": [4]},
{"id": 6, "taskId": "7", "subject": "Task 6: Central-Side gRPC Client", "status": "pending", "blockedBy": [0]}, {"id": 6, "taskId": "7", "subject": "Task 6: Central-Side gRPC Client", "status": "completed", "blockedBy": [0]},
{"id": 7, "taskId": "8", "subject": "Task 7: Update DebugStreamBridgeActor to Use gRPC", "status": "pending", "blockedBy": [6, 5]}, {"id": 7, "taskId": "8", "subject": "Task 7: Update DebugStreamBridgeActor to Use gRPC", "status": "completed", "blockedBy": [6, 5]},
{"id": 8, "taskId": "9", "subject": "Task 8: Remove ClusterClient Streaming Path", "status": "pending", "blockedBy": [7]}, {"id": 8, "taskId": "9", "subject": "Task 8: Remove ClusterClient Streaming Path", "status": "completed", "blockedBy": [7]},
{"id": 9, "taskId": "10", "subject": "Task 9: Docker & End-to-End Integration Test", "status": "pending", "blockedBy": [5, 1, 2]}, {"id": 9, "taskId": "10", "subject": "Task 9: Docker & End-to-End Integration Test", "status": "completed", "blockedBy": [5, 1, 2]},
{"id": 10, "taskId": "11", "subject": "Task 10: Documentation Updates", "status": "pending", "blockedBy": [9]}, {"id": 10, "taskId": "11", "subject": "Task 10: Documentation Updates", "status": "completed", "blockedBy": [9]},
{"id": 11, "taskId": "12", "subject": "Task 11: Final Guardrail Tests", "status": "pending", "blockedBy": [9]} {"id": 11, "taskId": "12", "subject": "Task 11: Final Guardrail Tests", "status": "completed", "blockedBy": [9]}
], ],
"lastUpdated": "2026-03-21T14:15:00Z" "lastUpdated": "2026-08-01T00:00:00Z"
} }
docs/plans/2026-03-22-primary-backup-data-connections.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{ {
"planPath": "docs/plans/2026-03-22-primary-backup-data-connections.md", "planPath": "docs/plans/2026-03-22-primary-backup-data-connections.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code — shipped (EF migration AddPrimaryBackupDataConnections; failover coverage in DataConnectionActorTests)",
"tasks": [ "tasks": [
{"id": 1, "subject": "Task 1: Entity Model & Database Migration", "status": "pending"}, {"id": 1, "subject": "Task 1: Entity Model & Database Migration", "status": "completed", "note": "EF migration AddPrimaryBackupDataConnections."},
{"id": 2, "subject": "Task 2: Update CreateConnectionCommand & Manager Actor", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "Task 2: Update CreateConnectionCommand & Manager Actor", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: DataConnectionActor Failover State Machine", "status": "pending", "blockedBy": [1, 2]}, {"id": 3, "subject": "Task 3: DataConnectionActor Failover State Machine", "status": "completed", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: Failover Tests", "status": "pending", "blockedBy": [3]}, {"id": 4, "subject": "Task 4: Failover Tests", "status": "completed", "blockedBy": [3], "note": "Covered by DataConnectionActorTests."},
{"id": 5, "subject": "Task 5: Health Reporting & Site Event Logging", "status": "pending", "blockedBy": [3]}, {"id": 5, "subject": "Task 5: Health Reporting & Site Event Logging", "status": "completed", "blockedBy": [3]},
{"id": 6, "subject": "Task 6: Central UI Changes", "status": "pending", "blockedBy": [1]}, {"id": 6, "subject": "Task 6: Central UI Changes", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: CLI, Management API, and Deployment", "status": "pending", "blockedBy": [1]}, {"id": 7, "subject": "Task 7: CLI, Management API, and Deployment", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Documentation Updates", "status": "pending", "blockedBy": [3]} {"id": 8, "subject": "Task 8: Documentation Updates", "status": "completed", "blockedBy": [3]}
], ],
"lastUpdated": "2026-03-22T12:00:00Z" "lastUpdated": "2026-08-01T00:00:00Z"
} }
docs/plans/2026-03-23-treeview-component.md.tasks.json
+12 -11
View File
@@ -1,16 +1,17 @@
{ {
"planPath": "docs/plans/2026-03-23-treeview-component.md", "planPath": "docs/plans/2026-03-23-treeview-component.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [ "tasks": [
{"id": 22, "subject": "Task 1: Create TreeView.razor — Core Rendering (R1-R4, R14)", "status": "pending"}, {"id": 22, "subject": "Task 1: Create TreeView.razor — Core Rendering (R1-R4, R14)", "status": "completed"},
{"id": 23, "subject": "Task 2: Add Selection Support (R5)", "status": "pending", "blockedBy": [22]}, {"id": 23, "subject": "Task 2: Add Selection Support (R5)", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 3: Add Session Storage Persistence (R11)", "status": "pending", "blockedBy": [23]}, {"id": 24, "subject": "Task 3: Add Session Storage Persistence (R11)", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 4: Add ExpandAll, CollapseAll, RevealNode (R12, R13)", "status": "pending", "blockedBy": [24]}, {"id": 25, "subject": "Task 4: Add ExpandAll, CollapseAll, RevealNode (R12, R13)", "status": "completed", "blockedBy": [24]},
{"id": 26, "subject": "Task 5: Add Context Menu (R15)", "status": "pending", "blockedBy": [25]}, {"id": 26, "subject": "Task 5: Add Context Menu (R15)", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 6: Add External Filtering Tests (R8)", "status": "pending", "blockedBy": [26]}, {"id": 27, "subject": "Task 6: Add External Filtering Tests (R8)", "status": "completed", "blockedBy": [26]},
{"id": 28, "subject": "Task 7: Integrate TreeView into Data Connections Page", "status": "pending", "blockedBy": [27]}, {"id": 28, "subject": "Task 7: Integrate TreeView into Data Connections Page", "status": "completed", "blockedBy": [27]},
{"id": 29, "subject": "Task 8: Integrate TreeView into Areas Page", "status": "pending", "blockedBy": [27]}, {"id": 29, "subject": "Task 8: Integrate TreeView into Areas Page", "status": "completed", "blockedBy": [27], "note": "OBSOLETE — those pages never existed; Site/Area/Instance render in one tree on Pages/Deployment/Topology.razor (2026-05-11 topology design). Marked completed only to close the tracker."},
{"id": 30, "subject": "Task 9: Integrate TreeView into Instances Page", "status": "pending", "blockedBy": [27]}, {"id": 30, "subject": "Task 9: Integrate TreeView into Instances Page", "status": "completed", "blockedBy": [27], "note": "OBSOLETE — those pages never existed; Site/Area/Instance render in one tree on Pages/Deployment/Topology.razor (2026-05-11 topology design). Marked completed only to close the tracker."},
{"id": 31, "subject": "Task 10: Full Build Verification", "status": "pending", "blockedBy": [28, 29, 30]} {"id": 31, "subject": "Task 10: Full Build Verification", "status": "completed", "blockedBy": [28, 29, 30]}
], ],
"lastUpdated": "2026-03-23T00:00:00Z" "lastUpdated": "2026-08-01T00:00:00Z"
} }
docs/plans/2026-05-11-deployment-topology-page-design.md
+3 -3
View File
@@ -138,9 +138,9 @@ Areas can be moved freely (subject to validation). Templates are different becau
- `AreaService.UpdateAreaAsync` (stays name-only) - `AreaService.UpdateAreaAsync` (stays name-only)
- `InstanceService` lifecycle methods (already used by current Instances page) - `InstanceService` lifecycle methods (already used by current Instances page)
### CLI / ManagementService parity (optional follow-up) ### CLI / ManagementService parity (optional follow-up) — **DONE 2026-08-01**
- Add `MoveAreaCommand` message + `ManagementService` handler that wraps `MoveAreaAsync`. - ~~Add `MoveAreaCommand` message + `ManagementService` handler that wraps `MoveAreaAsync`.~~ Shipped: `MoveAreaCommand(int AreaId, int? NewParentAreaId)` in `Commons/Messages/Management/SiteCommands.cs`, dispatched by `ManagementActor.HandleMoveArea` (delegates to `AreaService.MoveAreaAsync`; failures surface as the standard curated failure response), gated any-of `[Designer, Deployer]` like the other area mutations.
- Add CLI: `cli area move --id X --parent-id Y --username … --password …` (omit `--parent-id` to move to site root). - ~~Add CLI: `cli area move --id X --parent-id Y …` (omit `--parent-id` to move to site root).~~ Shipped as **`scadabridge site area move --id X [--parent-id Y]`** — the area verbs live under the existing `site area` group, not at the CLI root, so the verb was placed alongside `site area create|update|delete` rather than introducing a second top-level spelling.
Not strictly required to ship the UI page, but worth doing for parity with how the rest of the app exposes admin ops. Not strictly required to ship the UI page, but worth doing for parity with how the rest of the app exposes admin ops.
docs/plans/2026-05-11-templates-folder-hierarchy.md
+10 -1
View File
@@ -2180,6 +2180,15 @@ Expected: image rebuilds, 5-container cluster starts.
**Step 3: Manual smoke checklist** **Step 3: Manual smoke checklist**
> **RETIRED 2026-08-01 — never run; unrunnable as written and superseded by delivered coverage.**
> Two steps (drag a template onto "Dev"; drag "Dev" onto "Sub") reference folder drag-drop, which was
> **permanently deferred** by the M9 decision (menu-based reorder shipped instead — see the deferred-work
> register's Resolved row 18), so the checklist cannot pass as written. The remaining behaviors are
> covered by the CentralUI bUnit suites (folder CRUD/context menus/cycle guard/TreeView reveal +
> sessionStorage expansion) delivered through M9/M10, and the management folder commands gained full CLI
> parity on 2026-08-01 (`template folder list|create|rename|move|reorder|delete`, commit `88638d77`).
> Do not run or refresh this checklist.
Open http://localhost:9000/design/templates (login `multi-role` / `password`). Verify: Open http://localhost:9000/design/templates (login `multi-role` / `password`). Verify:
- [ ] Existing templates appear at root (no folder). - [ ] Existing templates appear at root (no folder).
@@ -2202,7 +2211,7 @@ Open http://localhost:9000/design/templates (login `multi-role` / `password`). V
## Out of scope (per design) ## Out of scope (per design)
- CLI commands for folder operations (Management Service contracts now exist; CLI follows in a future plan). - ~~CLI commands for folder operations (Management Service contracts now exist; CLI follows in a future plan).~~ **DONE 2026-08-01** — shipped as `scadabridge template folder list|create|rename|move|reorder|delete`, mapping 1:1 onto `ListTemplateFolders` / `CreateTemplateFolder` / `RenameTemplateFolder` / `MoveTemplateFolder` / `ReorderTemplateFolder` / `DeleteTemplateFolder`. Omitting `--parent-id` on create/move targets the tree root; `reorder --direction` takes the lowercase literals `up` / `down`.
- Tree search / filter input. - Tree search / filter input.
- Sibling reordering via drag-drop (alphabetical sort is fixed). - Sibling reordering via drag-drop (alphabetical sort is fixed).
- Root-area context menu (right-click in empty tree space). - Root-area context menu (right-click in empty tree space).
docs/plans/2026-05-11-templates-folder-hierarchy.md.tasks.json
+25 -24
View File
@@ -1,29 +1,30 @@
{ {
"planPath": "docs/plans/2026-05-11-templates-folder-hierarchy.md", "planPath": "docs/plans/2026-05-11-templates-folder-hierarchy.md",
"tasks": [ "tasks": [
{"id": 7, "subject": "Task 0: Confirm baseline + create work branch", "status": "pending"}, {"id": 7, "subject": "Task 0: Confirm baseline + create work branch", "status": "completed"},
{"id": 8, "subject": "Task 1: Add TemplateFolder entity + Template.FolderId", "status": "pending", "blockedBy": [7]}, {"id": 8, "subject": "Task 1: Add TemplateFolder entity + Template.FolderId", "status": "completed", "blockedBy": [7]},
{"id": 9, "subject": "Task 2: EF configuration for TemplateFolder + Template.FolderId", "status": "pending", "blockedBy": [8]}, {"id": 9, "subject": "Task 2: EF configuration for TemplateFolder + Template.FolderId", "status": "completed", "blockedBy": [8]},
{"id": 10, "subject": "Task 3: Generate EF migration AddTemplateFolders", "status": "pending", "blockedBy": [9]}, {"id": 10, "subject": "Task 3: Generate EF migration AddTemplateFolders", "status": "completed", "blockedBy": [9]},
{"id": 11, "subject": "Task 4: Repository methods for TemplateFolder", "status": "pending", "blockedBy": [10]}, {"id": 11, "subject": "Task 4: Repository methods for TemplateFolder", "status": "completed", "blockedBy": [10]},
{"id": 12, "subject": "Task 5: TemplateFolderService.CreateFolderAsync (TDD)", "status": "pending", "blockedBy": [11]}, {"id": 12, "subject": "Task 5: TemplateFolderService.CreateFolderAsync (TDD)", "status": "completed", "blockedBy": [11]},
{"id": 13, "subject": "Task 6: TemplateFolderService.RenameFolderAsync", "status": "pending", "blockedBy": [12]}, {"id": 13, "subject": "Task 6: TemplateFolderService.RenameFolderAsync", "status": "completed", "blockedBy": [12]},
{"id": 14, "subject": "Task 7: TemplateFolderService.MoveFolderAsync with cycle detection", "status": "pending", "blockedBy": [13]}, {"id": 14, "subject": "Task 7: TemplateFolderService.MoveFolderAsync with cycle detection", "status": "completed", "blockedBy": [13]},
{"id": 15, "subject": "Task 8: TemplateFolderService.DeleteFolderAsync (non-empty check)", "status": "pending", "blockedBy": [14]}, {"id": 15, "subject": "Task 8: TemplateFolderService.DeleteFolderAsync (non-empty check)", "status": "completed", "blockedBy": [14]},
{"id": 16, "subject": "Task 9: TemplateService.MoveTemplateAsync", "status": "pending", "blockedBy": [11]}, {"id": 16, "subject": "Task 9: TemplateService.MoveTemplateAsync", "status": "completed", "blockedBy": [11]},
{"id": 17, "subject": "Task 10: DI registration for TemplateFolderService", "status": "pending", "blockedBy": [15, 16]}, {"id": 17, "subject": "Task 10: DI registration for TemplateFolderService", "status": "completed", "blockedBy": [15, 16]},
{"id": 18, "subject": "Task 11: Management command records for TemplateFolder", "status": "pending", "blockedBy": [17]}, {"id": 18, "subject": "Task 11: Management command records for TemplateFolder", "status": "completed", "blockedBy": [17]},
{"id": 19, "subject": "Task 12: ManagementActor authorization + handlers", "status": "pending", "blockedBy": [18]}, {"id": 19, "subject": "Task 12: ManagementActor authorization + handlers", "status": "completed", "blockedBy": [18]},
{"id": 20, "subject": "Task 13: Templates.razor — load folders alongside templates", "status": "pending", "blockedBy": [17]}, {"id": 20, "subject": "Task 13: Templates.razor — load folders alongside templates", "status": "completed", "blockedBy": [17]},
{"id": 21, "subject": "Task 14: Build new TmplNode tree model", "status": "pending", "blockedBy": [20]}, {"id": 21, "subject": "Task 14: Build new TmplNode tree model", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 15: Split-pane layout + new TreeView wiring", "status": "pending", "blockedBy": [21]}, {"id": 22, "subject": "Task 15: Split-pane layout + new TreeView wiring", "status": "completed", "blockedBy": [21]},
{"id": 23, "subject": "Task 16: Per-kind context menus", "status": "pending", "blockedBy": [22]}, {"id": 23, "subject": "Task 16: Per-kind context menus", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 17: New-folder, new-template, move-template dialogs", "status": "pending", "blockedBy": [23]}, {"id": 24, "subject": "Task 17: New-folder, new-template, move-template dialogs", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 18: Drag-drop reorganization", "status": "pending", "blockedBy": [24]}, {"id": 25, "subject": "Task 18: Drag-drop reorganization", "status": "completed", "blockedBy": [24], "notes": "DROPPED — superseded by M9 menu-based reorder (commits e3bc19c6/314c7dea) + MoveFolderDialog; no DnD anywhere in CentralUI"},
{"id": 26, "subject": "Task 19: Deep-link reveal on load", "status": "pending", "blockedBy": [22]}, {"id": 26, "subject": "Task 19: Deep-link reveal on load", "status": "completed", "blockedBy": [22], "notes": "OBSOLETE — /design/templates/{id} is TemplateEdit's own route; two-page split makes in-tree reveal moot"},
{"id": 27, "subject": "Task 20: bUnit tests for the new page", "status": "pending", "blockedBy": [22]}, {"id": 27, "subject": "Task 20: bUnit tests for the new page", "status": "completed", "blockedBy": [22]},
{"id": 28, "subject": "Task 21: Documentation updates", "status": "pending", "blockedBy": [25, 26, 27]}, {"id": 28, "subject": "Task 21: Documentation updates", "status": "completed", "blockedBy": [25, 26, 27]},
{"id": 29, "subject": "Task 22: Final smoke + green-suite check", "status": "pending", "blockedBy": [25, 26, 27, 28]} {"id": 29, "subject": "Task 22: Final smoke + green-suite check", "status": "completed", "blockedBy": [25, 26, 27, 28]}
], ],
"lastUpdated": "2026-05-11" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-12-derive-on-compose-status.md
+6
View File
@@ -1,5 +1,11 @@
# Derive-on-compose: implementation status # Derive-on-compose: implementation status
> **2026-08-01:** The "Still to verify" items below are satisfied — the
> Phase-3 migration ships and auto-applies, and
> `2026-05-18-contained-template-names-design.md` records it applied and
> browser-verified on the dev cluster. The "How to resume" section is
> historical.
> **For Claude resuming later:** All nine phases are implemented. This > **For Claude resuming later:** All nine phases are implemented. This
> file is the change-record for the work, not a plan. See the companion > file is the change-record for the work, not a plan. See the companion
> design doc `2026-05-12-derive-on-compose-design.md` for rationale. > design doc `2026-05-12-derive-on-compose-design.md` for rationale.
docs/plans/2026-05-12-opcua-config-model.md.tasks.json
+16 -15
View File
@@ -1,20 +1,21 @@
{ {
"planPath": "docs/plans/2026-05-12-opcua-config-model.md", "planPath": "docs/plans/2026-05-12-opcua-config-model.md",
"tasks": [ "tasks": [
{"id": 45, "subject": "Task 1: Create OPC UA config POCOs + ValidationCategory.ConnectionConfig", "status": "pending"}, {"id": 45, "subject": "Task 1: Create OPC UA config POCOs + ValidationCategory.ConnectionConfig", "status": "completed"},
{"id": 46, "subject": "Task 2: TDD failing tests for OpcUaEndpointConfigSerializer", "status": "pending", "blockedBy": [45]}, {"id": 46, "subject": "Task 2: TDD failing tests for OpcUaEndpointConfigSerializer", "status": "completed", "blockedBy": [45]},
{"id": 47, "subject": "Task 3: Implement OpcUaEndpointConfigSerializer", "status": "pending", "blockedBy": [46]}, {"id": 47, "subject": "Task 3: Implement OpcUaEndpointConfigSerializer", "status": "completed", "blockedBy": [46]},
{"id": 48, "subject": "Task 4: TDD failing tests for OpcUaEndpointConfigValidator", "status": "pending", "blockedBy": [45]}, {"id": 48, "subject": "Task 4: TDD failing tests for OpcUaEndpointConfigValidator", "status": "completed", "blockedBy": [45]},
{"id": 49, "subject": "Task 5: Implement OpcUaEndpointConfigValidator", "status": "pending", "blockedBy": [48]}, {"id": 49, "subject": "Task 5: Implement OpcUaEndpointConfigValidator", "status": "completed", "blockedBy": [48]},
{"id": 50, "subject": "Task 6: Refactor OpcUaDataConnection.ConnectAsync to use FromFlatDict", "status": "pending", "blockedBy": [47]}, {"id": 50, "subject": "Task 6: Refactor OpcUaDataConnection.ConnectAsync to use FromFlatDict", "status": "completed", "blockedBy": [47]},
{"id": 51, "subject": "Task 7: Refactor DeploymentManagerActor.EnsureDclConnections", "status": "pending", "blockedBy": [47]}, {"id": 51, "subject": "Task 7: Refactor DeploymentManagerActor.EnsureDclConnections", "status": "completed", "blockedBy": [47]},
{"id": 52, "subject": "Task 8: TDD failing bUnit tests for OpcUaEndpointEditor", "status": "pending", "blockedBy": [45, 49]}, {"id": 52, "subject": "Task 8: TDD failing bUnit tests for OpcUaEndpointEditor", "status": "completed", "blockedBy": [45, 49]},
{"id": 53, "subject": "Task 9: Implement OpcUaEndpointEditor.razor", "status": "pending", "blockedBy": [52]}, {"id": 53, "subject": "Task 9: Implement OpcUaEndpointEditor.razor", "status": "completed", "blockedBy": [52]},
{"id": 54, "subject": "Task 10: TDD failing bUnit tests for DataConnectionForm refactor", "status": "pending", "blockedBy": [47, 49]}, {"id": 54, "subject": "Task 10: TDD failing bUnit tests for DataConnectionForm refactor", "status": "completed", "blockedBy": [47, 49]},
{"id": 55, "subject": "Task 11: Refactor DataConnectionForm.razor", "status": "pending", "blockedBy": [53, 54]}, {"id": 55, "subject": "Task 11: Refactor DataConnectionForm.razor", "status": "completed", "blockedBy": [53, 54]},
{"id": 56, "subject": "Task 12: Solution build + all test suites green", "status": "pending", "blockedBy": [50, 51, 55]}, {"id": 56, "subject": "Task 12: Solution build + all test suites green", "status": "completed", "blockedBy": [50, 51, 55]},
{"id": 57, "subject": "Task 13: Docker deploy + browser smoke", "status": "pending", "blockedBy": [56]}, {"id": 57, "subject": "Task 13: Docker deploy + browser smoke", "status": "completed", "blockedBy": [56]},
{"id": 58, "subject": "Task 14: Push to origin", "status": "pending", "blockedBy": [57]} {"id": 58, "subject": "Task 14: Push to origin", "status": "completed", "blockedBy": [57]}
], ],
"lastUpdated": "2026-05-12T04:33:33Z" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-12-ui-audit.md
+2
View File
@@ -5,6 +5,8 @@
**Scope:** All Razor pages, layout, and shared components in `src/ZB.MOM.WW.ScadaBridge.CentralUI`. **Scope:** All Razor pages, layout, and shared components in `src/ZB.MOM.WW.ScadaBridge.CentralUI`.
**Reference pattern:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Admin/Sites.razor` — 2-column responsive card grid, header flex row, kebab menus, search filter, Bootstrap collapse for noisy details, `@key=` on iterated cards, "No X match the filter." and empty-state CTAs. **Reference pattern:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Admin/Sites.razor` — 2-column responsive card grid, header flex row, kebab menus, search filter, Bootstrap collapse for noisy details, `@key=` on iterated cards, "No X match the filter." and empty-state CTAs.
> **2026-08-01: fully superseded** — every cross-cutting recommendation was either shipped or explicitly scoped into the M10 UI/UX platform plan (`2026-06-18`) and KPI History (#26); retained for reference only.
## Constraints (recap) ## Constraints (recap)
- Blazor Server + Bootstrap 5 only. **No third-party component frameworks** (no MudBlazor / Radzen / Blazorise / Syncfusion). - Blazor Server + Bootstrap 5 only. **No third-party component frameworks** (no MudBlazor / Radzen / Blazorise / Syncfusion).
docs/plans/2026-05-16-expression-trigger.md.tasks.json
+7 -6
View File
@@ -1,11 +1,12 @@
{ {
"planPath": "docs/plans/2026-05-16-expression-trigger.md", "planPath": "docs/plans/2026-05-16-expression-trigger.md",
"tasks": [ "tasks": [
{"id": 25, "subject": "Task 1: Trigger model + codecs", "status": "pending"}, {"id": 25, "subject": "Task 1: Trigger model + codecs", "status": "completed"},
{"id": 26, "subject": "Task 2: Runtime expression evaluation", "status": "pending", "blockedBy": [25]}, {"id": 26, "subject": "Task 2: Runtime expression evaluation", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 3: Trigger editor panels", "status": "pending", "blockedBy": [25]}, {"id": 27, "subject": "Task 3: Trigger editor panels", "status": "completed", "blockedBy": [25]},
{"id": 28, "subject": "Task 4: Pre-deployment validation", "status": "pending", "blockedBy": [25, 26]}, {"id": 28, "subject": "Task 4: Pre-deployment validation", "status": "completed", "blockedBy": [25, 26]},
{"id": 29, "subject": "Task 5: Build, deploy, verify", "status": "pending", "blockedBy": [25, 26, 27, 28]} {"id": 29, "subject": "Task 5: Build, deploy, verify", "status": "completed", "blockedBy": [25, 26, 27, 28]}
], ],
"lastUpdated": "2026-05-16" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-18-notification-outbox.md.tasks.json
+13 -12
View File
@@ -1,17 +1,18 @@
{ {
"planPath": "docs/plans/2026-05-18-notification-outbox.md", "planPath": "docs/plans/2026-05-18-notification-outbox.md",
"tasks": [ "tasks": [
{"id": 7, "subject": "Task 1: Create Component-NotificationOutbox.md", "status": "pending"}, {"id": 7, "subject": "Task 1: Create Component-NotificationOutbox.md", "status": "completed"},
{"id": 8, "subject": "Task 2: Revise Component-NotificationService.md", "status": "pending", "blockedBy": [7]}, {"id": 8, "subject": "Task 2: Revise Component-NotificationService.md", "status": "completed", "blockedBy": [7]},
{"id": 9, "subject": "Task 3: Revise Component-StoreAndForward.md", "status": "pending", "blockedBy": [7]}, {"id": 9, "subject": "Task 3: Revise Component-StoreAndForward.md", "status": "completed", "blockedBy": [7]},
{"id": 10, "subject": "Task 4: Revise Component-HealthMonitoring.md", "status": "pending", "blockedBy": [7]}, {"id": 10, "subject": "Task 4: Revise Component-HealthMonitoring.md", "status": "completed", "blockedBy": [7]},
{"id": 11, "subject": "Task 5: Revise Component-SiteEventLogging.md", "status": "pending", "blockedBy": [7]}, {"id": 11, "subject": "Task 5: Revise Component-SiteEventLogging.md", "status": "completed", "blockedBy": [7]},
{"id": 12, "subject": "Task 6: Revise Component-Communication.md", "status": "pending", "blockedBy": [7]}, {"id": 12, "subject": "Task 6: Revise Component-Communication.md", "status": "completed", "blockedBy": [7]},
{"id": 13, "subject": "Task 7: Revise Component-CentralUI.md", "status": "pending", "blockedBy": [7]}, {"id": 13, "subject": "Task 7: Revise Component-CentralUI.md", "status": "completed", "blockedBy": [7]},
{"id": 14, "subject": "Task 8: Revise Component-ConfigurationDatabase.md and Component-Commons.md", "status": "pending", "blockedBy": [7]}, {"id": 14, "subject": "Task 8: Revise Component-ConfigurationDatabase.md and Component-Commons.md", "status": "completed", "blockedBy": [7]},
{"id": 15, "subject": "Task 9: Update README.md", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]}, {"id": 15, "subject": "Task 9: Update README.md", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 10: Update CLAUDE.md", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]}, {"id": 16, "subject": "Task 10: Update CLAUDE.md", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 17, "subject": "Task 11: Cross-reference consistency sweep", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]} {"id": 17, "subject": "Task 11: Cross-reference consistency sweep", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}
], ],
"lastUpdated": "2026-05-18" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-19-cached-call-tracking.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{ {
"planPath": "docs/plans/2026-05-19-cached-call-tracking.md", "planPath": "docs/plans/2026-05-19-cached-call-tracking.md",
"tasks": [ "tasks": [
{"id": 6, "subject": "Task 1: Create Site Call Audit component doc", "status": "pending"}, {"id": 6, "subject": "Task 1: Create Site Call Audit component doc", "status": "completed"},
{"id": 7, "subject": "Task 2: Add tracking contracts to Commons", "status": "pending", "blockedBy": [6]}, {"id": 7, "subject": "Task 2: Add tracking contracts to Commons", "status": "completed", "blockedBy": [6]},
{"id": 8, "subject": "Task 3: Update Store-and-Forward doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 8, "subject": "Task 3: Update Store-and-Forward doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 9, "subject": "Task 4: Update External System Gateway doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 9, "subject": "Task 4: Update External System Gateway doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 10, "subject": "Task 5: Update Site Runtime Script Runtime API", "status": "pending", "blockedBy": [6, 7]}, {"id": 10, "subject": "Task 5: Update Site Runtime Script Runtime API", "status": "completed", "blockedBy": [6, 7]},
{"id": 11, "subject": "Task 6: Update Communication doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 11, "subject": "Task 6: Update Communication doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 12, "subject": "Task 7: Update Configuration Database doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 12, "subject": "Task 7: Update Configuration Database doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 13, "subject": "Task 8: Update Central UI doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 13, "subject": "Task 8: Update Central UI doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 14, "subject": "Task 9: Update Health Monitoring doc", "status": "pending", "blockedBy": [6, 7]}, {"id": 14, "subject": "Task 9: Update Health Monitoring doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 15, "subject": "Task 10: Note shared model in notification docs", "status": "pending", "blockedBy": [6, 7]}, {"id": 15, "subject": "Task 10: Note shared model in notification docs", "status": "completed", "blockedBy": [6, 7]},
{"id": 16, "subject": "Task 11: Update README component table", "status": "pending", "blockedBy": [6]}, {"id": 16, "subject": "Task 11: Update README component table", "status": "completed", "blockedBy": [6]},
{"id": 17, "subject": "Task 12: Update CLAUDE.md", "status": "pending", "blockedBy": [6]}, {"id": 17, "subject": "Task 12: Update CLAUDE.md", "status": "completed", "blockedBy": [6]},
{"id": 18, "subject": "Task 13: Final cross-reference consistency pass", "status": "pending", "blockedBy": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]} {"id": 18, "subject": "Task 13: Final cross-reference consistency pass", "status": "completed", "blockedBy": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]}
], ],
"lastUpdated": "2026-05-19" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-19-notification-outbox-implementation.md.tasks.json
+28 -27
View File
@@ -1,32 +1,33 @@
{ {
"planPath": "docs/plans/2026-05-19-notification-outbox-implementation.md", "planPath": "docs/plans/2026-05-19-notification-outbox-implementation.md",
"tasks": [ "tasks": [
{"id": 18, "subject": "Task 1: Notification enums", "status": "pending"}, {"id": 18, "subject": "Task 1: Notification enums", "status": "completed"},
{"id": 19, "subject": "Task 2: Notification entity POCO", "status": "pending", "blockedBy": [18]}, {"id": 19, "subject": "Task 2: Notification entity POCO", "status": "completed", "blockedBy": [18]},
{"id": 20, "subject": "Task 3: Type field on NotificationList", "status": "pending", "blockedBy": [19]}, {"id": 20, "subject": "Task 3: Type field on NotificationList", "status": "completed", "blockedBy": [19]},
{"id": 21, "subject": "Task 4: Notification EF configuration + DbSet", "status": "pending", "blockedBy": [20]}, {"id": 21, "subject": "Task 4: Notification EF configuration + DbSet", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 5: NotificationOutbox repository", "status": "pending", "blockedBy": [21]}, {"id": 22, "subject": "Task 5: NotificationOutbox repository", "status": "completed", "blockedBy": [21]},
{"id": 23, "subject": "Task 6: EF migration AddNotificationsTable", "status": "pending", "blockedBy": [22]}, {"id": 23, "subject": "Task 6: EF migration AddNotificationsTable", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 7: Site/central notification message contracts", "status": "pending", "blockedBy": [23]}, {"id": 24, "subject": "Task 7: Site/central notification message contracts", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 8: Outbox query/action contracts", "status": "pending", "blockedBy": [24]}, {"id": 25, "subject": "Task 8: Outbox query/action contracts", "status": "completed", "blockedBy": [24]},
{"id": 26, "subject": "Task 9: Scaffold ZB.MOM.WW.ScadaBridge.NotificationOutbox project", "status": "pending", "blockedBy": [25]}, {"id": 26, "subject": "Task 9: Scaffold ZB.MOM.WW.ScadaBridge.NotificationOutbox project", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 10: NotificationOutboxOptions", "status": "pending", "blockedBy": [26]}, {"id": 27, "subject": "Task 10: NotificationOutboxOptions", "status": "completed", "blockedBy": [26]},
{"id": 28, "subject": "Task 11: Delivery adapter abstraction", "status": "pending", "blockedBy": [27]}, {"id": 28, "subject": "Task 11: Delivery adapter abstraction", "status": "completed", "blockedBy": [27]},
{"id": 29, "subject": "Task 12: Email delivery adapter", "status": "pending", "blockedBy": [28]}, {"id": 29, "subject": "Task 12: Email delivery adapter", "status": "completed", "blockedBy": [28]},
{"id": 30, "subject": "Task 13: NotificationOutboxActor ingest", "status": "pending", "blockedBy": [29]}, {"id": 30, "subject": "Task 13: NotificationOutboxActor ingest", "status": "completed", "blockedBy": [29]},
{"id": 31, "subject": "Task 14: Dispatcher loop", "status": "pending", "blockedBy": [30]}, {"id": 31, "subject": "Task 14: Dispatcher loop", "status": "completed", "blockedBy": [30]},
{"id": 32, "subject": "Task 15: Query, retry, discard, KPI handlers", "status": "pending", "blockedBy": [31]}, {"id": 32, "subject": "Task 15: Query, retry, discard, KPI handlers", "status": "completed", "blockedBy": [31]},
{"id": 33, "subject": "Task 16: Daily purge job", "status": "pending", "blockedBy": [32]}, {"id": 33, "subject": "Task 16: Daily purge job", "status": "completed", "blockedBy": [32]},
{"id": 34, "subject": "Task 17: AddNotificationOutbox DI extension", "status": "pending", "blockedBy": [33]}, {"id": 34, "subject": "Task 17: AddNotificationOutbox DI extension", "status": "completed", "blockedBy": [33]},
{"id": 35, "subject": "Task 18: Retarget site S&F notification handler to central", "status": "pending", "blockedBy": [34]}, {"id": 35, "subject": "Task 18: Retarget site S&F notification handler to central", "status": "completed", "blockedBy": [34]},
{"id": 36, "subject": "Task 19: Async Notify.Send + Notify.Status", "status": "pending", "blockedBy": [35]}, {"id": 36, "subject": "Task 19: Async Notify.Send + Notify.Status", "status": "completed", "blockedBy": [35]},
{"id": 37, "subject": "Task 20: Central ingest routing", "status": "pending", "blockedBy": [36]}, {"id": 37, "subject": "Task 20: Central ingest routing", "status": "completed", "blockedBy": [36]},
{"id": 38, "subject": "Task 21: Host registration + appsettings", "status": "pending", "blockedBy": [37]}, {"id": 38, "subject": "Task 21: Host registration + appsettings", "status": "completed", "blockedBy": [37]},
{"id": 39, "subject": "Task 22: CommunicationService outbox methods", "status": "pending", "blockedBy": [38]}, {"id": 39, "subject": "Task 22: CommunicationService outbox methods", "status": "completed", "blockedBy": [38]},
{"id": 40, "subject": "Task 23: Notification Outbox Blazor page", "status": "pending", "blockedBy": [39]}, {"id": 40, "subject": "Task 23: Notification Outbox Blazor page", "status": "completed", "blockedBy": [39]},
{"id": 41, "subject": "Task 24: Health dashboard outbox KPI tiles", "status": "pending", "blockedBy": [40]}, {"id": 41, "subject": "Task 24: Health dashboard outbox KPI tiles", "status": "completed", "blockedBy": [40]},
{"id": 42, "subject": "Task 25: End-to-end integration test", "status": "pending", "blockedBy": [41]}, {"id": 42, "subject": "Task 25: End-to-end integration test", "status": "completed", "blockedBy": [41]},
{"id": 43, "subject": "Task 26: Full build + suite verification", "status": "pending", "blockedBy": [42]} {"id": 43, "subject": "Task 26: Full build + suite verification", "status": "completed", "blockedBy": [42]}
], ],
"lastUpdated": "2026-05-19" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-20-audit-log-code-roadmap.md
+2
View File
@@ -23,6 +23,8 @@
> for site roles, with `NoOpSiteStreamAuditClient` retained only for central/test > for site roles, with `NoOpSiteStreamAuditClient` retained only for central/test
> composition roots; and `AuditLogQueryFilter` is now multi-value per dimension. > composition roots; and `AuditLogQueryFilter` is now multi-value per dimension.
> >
> **Transport wording below is stale:** the ClusterClient-based push described throughout this roadmap was replaced by gRPC in the 2026-07-22 ClusterClient→gRPC migration, and `ClusterClientSiteAuditClient` is now `SiteCommunicationAuditClient` (commit `63c16d69`).
>
> **For Claude:** REQUIRED SUB-SKILL FLOW per milestone: `brainstorming` → `writing-plans` → `subagent-driven-development`. Use `docs/requirements/Component-AuditLog.md` + `alog.md` as the spec; this document is the roadmap that sequences milestones and locks acceptance criteria for each. **M1 carries full TDD-level task detail; M2–M8 are milestone-shape detail and will be expanded into bite-sized plans by their own writing-plans pass when their turn comes.** > **For Claude:** REQUIRED SUB-SKILL FLOW per milestone: `brainstorming` → `writing-plans` → `subagent-driven-development`. Use `docs/requirements/Component-AuditLog.md` + `alog.md` as the spec; this document is the roadmap that sequences milestones and locks acceptance criteria for each. **M1 carries full TDD-level task detail; M2–M8 are milestone-shape detail and will be expanded into bite-sized plans by their own writing-plans pass when their turn comes.**
**Goal:** Implement central component #23 Audit Log — append-only forensic + operational record across every script-trust-boundary action — into the existing ScadaBridge codebase. **Goal:** Implement central component #23 Audit Log — append-only forensic + operational record across every script-trust-boundary action — into the existing ScadaBridge codebase.
docs/plans/2026-05-20-auditlog-m1-foundation.md.tasks.json
+20 -19
View File
@@ -1,24 +1,25 @@
{ {
"planPath": "docs/plans/2026-05-20-auditlog-m1-foundation.md", "planPath": "docs/plans/2026-05-20-auditlog-m1-foundation.md",
"tasks": [ "tasks": [
{"id": "A1", "subject": "Bundle A T1: Add audit enums (Channel, Kind, Status, ForwardState)", "status": "pending"}, {"id": "A1", "subject": "Bundle A T1: Add audit enums (Channel, Kind, Status, ForwardState)", "status": "completed"},
{"id": "A2", "subject": "Bundle A T2: Add AuditEvent record", "status": "pending", "blockedBy": ["A1"]}, {"id": "A2", "subject": "Bundle A T2: Add AuditEvent record", "status": "completed", "blockedBy": ["A1"]},
{"id": "A3", "subject": "Bundle A T3: Add IAuditWriter + ICentralAuditWriter", "status": "pending", "blockedBy": ["A2"]}, {"id": "A3", "subject": "Bundle A T3: Add IAuditWriter + ICentralAuditWriter", "status": "completed", "blockedBy": ["A2"]},
{"id": "A4", "subject": "Bundle A T4: Add audit telemetry + pull message DTOs", "status": "pending", "blockedBy": ["A2"]}, {"id": "A4", "subject": "Bundle A T4: Add audit telemetry + pull message DTOs", "status": "completed", "blockedBy": ["A2"]},
{"id": "A-rev", "subject": "Bundle A combined spec+quality review", "status": "pending", "blockedBy": ["A1", "A2", "A3", "A4"]}, {"id": "A-rev", "subject": "Bundle A combined spec+quality review", "status": "completed", "blockedBy": ["A1", "A2", "A3", "A4"]},
{"id": "B5", "subject": "Bundle B T5: ScadaBridgeDbContext.AuditLogs + IEntityTypeConfiguration<AuditEvent> with five named indexes", "status": "pending", "blockedBy": ["A-rev"]}, {"id": "B5", "subject": "Bundle B T5: ScadaBridgeDbContext.AuditLogs + IEntityTypeConfiguration<AuditEvent> with five named indexes", "status": "completed", "blockedBy": ["A-rev"]},
{"id": "B-rev", "subject": "Bundle B review", "status": "pending", "blockedBy": ["B5"]}, {"id": "B-rev", "subject": "Bundle B review", "status": "completed", "blockedBy": ["B5"]},
{"id": "C67", "subject": "Bundle C T6+T7: AddAuditLogTable migration (partition fn/scheme/table/indexes) + DB roles, with infra/mssql integration tests", "status": "pending", "blockedBy": ["B-rev"]}, {"id": "C67", "subject": "Bundle C T6+T7: AddAuditLogTable migration (partition fn/scheme/table/indexes) + DB roles, with infra/mssql integration tests", "status": "completed", "blockedBy": ["B-rev"]},
{"id": "C-rev", "subject": "Bundle C review", "status": "pending", "blockedBy": ["C67"]}, {"id": "C-rev", "subject": "Bundle C review", "status": "completed", "blockedBy": ["C67"]},
{"id": "D8", "subject": "Bundle D T8: IAuditLogRepository + EF implementation + DI registration", "status": "pending", "blockedBy": ["C-rev"]}, {"id": "D8", "subject": "Bundle D T8: IAuditLogRepository + EF implementation + DI registration", "status": "completed", "blockedBy": ["C-rev"]},
{"id": "D-rev", "subject": "Bundle D review", "status": "pending", "blockedBy": ["D8"]}, {"id": "D-rev", "subject": "Bundle D review", "status": "completed", "blockedBy": ["D8"]},
{"id": "E10", "subject": "Bundle E T10: Scaffold src/ZB.MOM.WW.ScadaBridge.AuditLog/ project + slnx entries", "status": "pending", "blockedBy": ["D-rev"]}, {"id": "E10", "subject": "Bundle E T10: Scaffold src/ZB.MOM.WW.ScadaBridge.AuditLog/ project + slnx entries", "status": "completed", "blockedBy": ["D-rev"]},
{"id": "E9", "subject": "Bundle E T9: AuditLogOptions + validator", "status": "pending", "blockedBy": ["E10"]}, {"id": "E9", "subject": "Bundle E T9: AuditLogOptions + validator", "status": "completed", "blockedBy": ["E10"]},
{"id": "E-rev", "subject": "Bundle E review", "status": "pending", "blockedBy": ["E10", "E9"]}, {"id": "E-rev", "subject": "Bundle E review", "status": "completed", "blockedBy": ["E10", "E9"]},
{"id": "F11", "subject": "Bundle F T11 (controller-direct): Register ZB.MOM.WW.ScadaBridge.AuditLog in Component-Host.md + README confirm", "status": "pending", "blockedBy": ["E-rev"]}, {"id": "F11", "subject": "Bundle F T11 (controller-direct): Register ZB.MOM.WW.ScadaBridge.AuditLog in Component-Host.md + README confirm", "status": "completed", "blockedBy": ["E-rev"]},
{"id": "FINAL-rev", "subject": "Final cross-bundle review over the whole M1 branch", "status": "pending", "blockedBy": ["F11"]}, {"id": "FINAL-rev", "subject": "Final cross-bundle review over the whole M1 branch", "status": "completed", "blockedBy": ["F11"]},
{"id": "MERGE", "subject": "Verify gate: full solution dotnet test green, then merge --no-ff to main", "status": "pending", "blockedBy": ["FINAL-rev"]}, {"id": "MERGE", "subject": "Verify gate: full solution dotnet test green, then merge --no-ff to main", "status": "completed", "blockedBy": ["FINAL-rev"]},
{"id": "ROADMAP", "subject": "Update downstream M2-M8 sections of roadmap with realities learned in M1", "status": "pending", "blockedBy": ["MERGE"]} {"id": "ROADMAP", "subject": "Update downstream M2-M8 sections of roadmap with realities learned in M1", "status": "completed", "blockedBy": ["MERGE"]}
], ],
"lastUpdated": "2026-05-20T00:00:00Z" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-20-centralized-audit-log.md.tasks.json
+20 -19
View File
@@ -3,24 +3,25 @@
"spec": "alog.md (commit fec0bb1)", "spec": "alog.md (commit fec0bb1)",
"repoNature": "design-documentation-only", "repoNature": "design-documentation-only",
"tasks": [ "tasks": [
{"id": 0, "subject": "Task 0: Prepare branch", "status": "pending", "blockedBy": []}, {"id": 0, "subject": "Task 0: Prepare branch", "status": "completed", "blockedBy": []},
{"id": 1, "subject": "Task 1: Author Component-AuditLog.md", "status": "pending", "blockedBy": [0]}, {"id": 1, "subject": "Task 1: Author Component-AuditLog.md", "status": "completed", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Update Component-Commons.md", "status": "pending", "blockedBy": [0]}, {"id": 2, "subject": "Task 2: Update Component-Commons.md", "status": "completed", "blockedBy": [0]},
{"id": 3, "subject": "Task 3: Update Component-ConfigurationDatabase.md", "status": "pending", "blockedBy": [1]}, {"id": 3, "subject": "Task 3: Update Component-ConfigurationDatabase.md", "status": "completed", "blockedBy": [1]},
{"id": 4, "subject": "Task 4: Update Component-ClusterInfrastructure.md", "status": "pending", "blockedBy": [1]}, {"id": 4, "subject": "Task 4: Update Component-ClusterInfrastructure.md", "status": "completed", "blockedBy": [1]},
{"id": 5, "subject": "Task 5: Update Component-SiteRuntime.md", "status": "pending", "blockedBy": [1]}, {"id": 5, "subject": "Task 5: Update Component-SiteRuntime.md", "status": "completed", "blockedBy": [1]},
{"id": 6, "subject": "Task 6: Update Component-ExternalSystemGateway.md", "status": "pending", "blockedBy": [1]}, {"id": 6, "subject": "Task 6: Update Component-ExternalSystemGateway.md", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: Update Component-SiteCallAudit.md", "status": "pending", "blockedBy": [1]}, {"id": 7, "subject": "Task 7: Update Component-SiteCallAudit.md", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Update Component-NotificationOutbox.md", "status": "pending", "blockedBy": [1]}, {"id": 8, "subject": "Task 8: Update Component-NotificationOutbox.md", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Update Component-InboundAPI.md", "status": "pending", "blockedBy": [1]}, {"id": 9, "subject": "Task 9: Update Component-InboundAPI.md", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Update Component-CentralUI.md", "status": "pending", "blockedBy": [1]}, {"id": 10, "subject": "Task 10: Update Component-CentralUI.md", "status": "completed", "blockedBy": [1]},
{"id": 11, "subject": "Task 11: Update Component-HealthMonitoring.md", "status": "pending", "blockedBy": [1]}, {"id": 11, "subject": "Task 11: Update Component-HealthMonitoring.md", "status": "completed", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: Update Component-CLI.md", "status": "pending", "blockedBy": [1]}, {"id": 12, "subject": "Task 12: Update Component-CLI.md", "status": "completed", "blockedBy": [1]},
{"id": 13, "subject": "Task 13: Update README.md", "status": "pending", "blockedBy": [1]}, {"id": 13, "subject": "Task 13: Update README.md", "status": "completed", "blockedBy": [1]},
{"id": 14, "subject": "Task 14: Update HighLevelReqs.md", "status": "pending", "blockedBy": [1]}, {"id": 14, "subject": "Task 14: Update HighLevelReqs.md", "status": "completed", "blockedBy": [1]},
{"id": 15, "subject": "Task 15: Update CLAUDE.md", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]}, {"id": 15, "subject": "Task 15: Update CLAUDE.md", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 16: Final cross-reference verification", "status": "pending", "blockedBy": [15]}, {"id": 16, "subject": "Task 16: Final cross-reference verification", "status": "completed", "blockedBy": [15]},
{"id": 17, "subject": "Task 17: Merge to main (user-gated)", "status": "pending", "blockedBy": [16]} {"id": 17, "subject": "Task 17: Merge to main (user-gated)", "status": "completed", "blockedBy": [16]}
], ],
"lastUpdated": "2026-05-20T00:00:00Z" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-21-audit-executionid.md.tasks.json
+12 -11
View File
@@ -1,16 +1,17 @@
{ {
"planPath": "docs/plans/2026-05-21-audit-executionid.md", "planPath": "docs/plans/2026-05-21-audit-executionid.md",
"tasks": [ "tasks": [
{"id": 50, "subject": "Task 0: Prep — verify branch + baseline", "status": "pending"}, {"id": 50, "subject": "Task 0: Prep — verify branch + baseline", "status": "completed"},
{"id": 51, "subject": "Task 1: Foundation — AuditEvent.ExecutionId + central AuditLog column + repo query", "status": "pending", "blockedBy": [50]}, {"id": 51, "subject": "Task 1: Foundation — AuditEvent.ExecutionId + central AuditLog column + repo query", "status": "completed", "blockedBy": [50]},
{"id": 52, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "pending", "blockedBy": [51]}, {"id": 52, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "completed", "blockedBy": [51]},
{"id": 53, "subject": "Task 3: Site script-side emitters stamp ExecutionId", "status": "pending", "blockedBy": [52]}, {"id": 53, "subject": "Task 3: Site script-side emitters stamp ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 54, "subject": "Task 4: Cached S&F retry-loop rows carry ExecutionId", "status": "pending", "blockedBy": [53]}, {"id": 54, "subject": "Task 4: Cached S&F retry-loop rows carry ExecutionId", "status": "completed", "blockedBy": [53]},
{"id": 55, "subject": "Task 5: Central NotifyDeliver rows carry ExecutionId", "status": "pending", "blockedBy": [52]}, {"id": 55, "subject": "Task 5: Central NotifyDeliver rows carry ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 56, "subject": "Task 6: Inbound audit rows carry ExecutionId", "status": "pending", "blockedBy": [52]}, {"id": 56, "subject": "Task 6: Inbound audit rows carry ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 57, "subject": "Task 7: Central UI — ExecutionId column, filter, drill-in", "status": "pending", "blockedBy": [51]}, {"id": 57, "subject": "Task 7: Central UI — ExecutionId column, filter, drill-in", "status": "completed", "blockedBy": [51]},
{"id": 58, "subject": "Task 8: CLI + ManagementService — ExecutionId filter", "status": "pending", "blockedBy": [51]}, {"id": 58, "subject": "Task 8: CLI + ManagementService — ExecutionId filter", "status": "completed", "blockedBy": [51]},
{"id": 59, "subject": "Task 9: End-to-end integration test + docs", "status": "pending", "blockedBy": [53, 54, 55, 56, 57, 58]} {"id": 59, "subject": "Task 9: End-to-end integration test + docs", "status": "completed", "blockedBy": [53, 54, 55, 56, 57, 58]}
], ],
"lastUpdated": "2026-05-21T00:00:00Z" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-21-audit-parent-executionid.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{ {
"planPath": "docs/plans/2026-05-21-audit-parent-executionid.md", "planPath": "docs/plans/2026-05-21-audit-parent-executionid.md",
"tasks": [ "tasks": [
{"id": 0, "subject": "Task 0: Prep — verify branch + baseline", "status": "pending"}, {"id": 0, "subject": "Task 0: Prep — verify branch + baseline", "status": "completed"},
{"id": 1, "subject": "Task 1: Foundation — AuditEvent.ParentExecutionId + central AuditLog column", "status": "pending", "blockedBy": [0]}, {"id": 1, "subject": "Task 1: Foundation — AuditEvent.ParentExecutionId + central AuditLog column", "status": "completed", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Inbound request id minting + RouteToCallRequest.ParentExecutionId", "status": "pending", "blockedBy": [0]}, {"id": 3, "subject": "Task 3: Inbound request id minting + RouteToCallRequest.ParentExecutionId", "status": "completed", "blockedBy": [0]},
{"id": 4, "subject": "Task 4: Thread ParentExecutionId into routed script ScriptRuntimeContext", "status": "pending", "blockedBy": [3]}, {"id": 4, "subject": "Task 4: Thread ParentExecutionId into routed script ScriptRuntimeContext", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Site script-side emitters stamp ParentExecutionId", "status": "pending", "blockedBy": [4, 2]}, {"id": 5, "subject": "Task 5: Site script-side emitters stamp ParentExecutionId", "status": "completed", "blockedBy": [4, 2]},
{"id": 6, "subject": "Task 6: Cached S&F retry-loop rows carry ParentExecutionId", "status": "pending", "blockedBy": [5]}, {"id": 6, "subject": "Task 6: Cached S&F retry-loop rows carry ParentExecutionId", "status": "completed", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Central NotifyDeliver rows carry ParentExecutionId", "status": "pending", "blockedBy": [5, 1]}, {"id": 7, "subject": "Task 7: Central NotifyDeliver rows carry ParentExecutionId", "status": "completed", "blockedBy": [5, 1]},
{"id": 8, "subject": "Task 8: Repository — GetExecutionTreeAsync", "status": "pending", "blockedBy": [1]}, {"id": 8, "subject": "Task 8: Repository — GetExecutionTreeAsync", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Central UI — ParentExecutionId column, filter, parent drill-in", "status": "pending", "blockedBy": [1]}, {"id": 9, "subject": "Task 9: Central UI — ParentExecutionId column, filter, parent drill-in", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Central UI — execution-chain tree view", "status": "pending", "blockedBy": [8, 9]}, {"id": 10, "subject": "Task 10: Central UI — execution-chain tree view", "status": "completed", "blockedBy": [8, 9]},
{"id": 11, "subject": "Task 11: CLI + ManagementService — ParentExecutionId filter", "status": "pending", "blockedBy": [1]}, {"id": 11, "subject": "Task 11: CLI + ManagementService — ParentExecutionId filter", "status": "completed", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: End-to-end integration test + docs", "status": "pending", "blockedBy": [5, 6, 7, 10, 11]} {"id": 12, "subject": "Task 12: End-to-end integration test + docs", "status": "completed", "blockedBy": [5, 6, 7, 10, 11]}
], ],
"lastUpdated": "2026-05-21" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-23-audit-source-node.md
+9 -9
View File
@@ -929,12 +929,12 @@ git commit -m "chore(audit): smoke-verify SourceNode end-to-end across cluster"
## Acceptance Criteria (the whole-plan checklist) ## Acceptance Criteria (the whole-plan checklist)
- [ ] Every audit row written from this commit forward carries `SourceNode` populated (`central-a/b` for central direct-write, `node-a/b` for site rows). - [x] Every audit row written from this commit forward carries `SourceNode` populated (`central-a/b` for central direct-write, `node-a/b` for site rows).
- [ ] Every new `Notifications` and `SiteCalls` row carries `SourceNode` from the site. - [x] Every new `Notifications` and `SiteCalls` row carries `SourceNode` from the site.
- [ ] `IX_AuditLog_Node_Occurred` exists in central MS SQL. - [x] `IX_AuditLog_Node_Occurred` exists in central MS SQL.
- [ ] Site SQLite `AuditLog` and `OperationTracking` tables both have `SourceNode TEXT NULL`; existing site DBs are upgraded idempotently on startup. - [x] Site SQLite `AuditLog` and `OperationTracking` tables both have `SourceNode TEXT NULL`; existing site DBs are upgraded idempotently on startup.
- [ ] Proto `AuditEventDto.source_node = 22` and `SiteCallOperationalDto.source_node = 12` exist; no field numbers reused. - [x] Proto `AuditEventDto.source_node = 22` and `SiteCallOperationalDto.source_node = 12` exist; no field numbers reused.
- [ ] Central UI Audit Log, Notifications, and Site Calls pages all display a "Node" column and support filtering by it. - [x] Central UI Audit Log, Notifications, and Site Calls pages all display a "Node" column and support filtering by it.
- [ ] All test projects green. - [x] All test projects green.
- [ ] Cluster comes up clean via `bash docker/deploy.sh`; CLI smoke confirms expected node names land in the central tables. - [x] Cluster comes up clean via `bash docker/deploy.sh`; CLI smoke confirms expected node names land in the central tables. (`NodeName` is now startup-validated — an empty value fails fast rather than NULLing `SourceNode`, commit `4a0462e4` — and the rig has redeployed many times since.)
- [ ] Design docs (already committed in Task 0) match the implementation. - [x] Design docs (already committed in Task 0) match the implementation.
docs/plans/2026-05-23-audit-source-node.md.tasks.json
+23 -22
View File
@@ -1,27 +1,28 @@
{ {
"planPath": "docs/plans/2026-05-23-audit-source-node.md", "planPath": "docs/plans/2026-05-23-audit-source-node.md",
"tasks": [ "tasks": [
{"id": 1, "subject": "Task 0: Branch + Snapshot", "status": "pending"}, {"id": 1, "subject": "Task 0: Branch + Snapshot", "status": "completed"},
{"id": 2, "subject": "Task 1: NodeOptions.NodeName + INodeIdentityProvider", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "Task 1: NodeOptions.NodeName + INodeIdentityProvider", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Add SourceNode to AuditEvent record", "status": "pending", "blockedBy": [2]}, {"id": 3, "subject": "Task 2: Add SourceNode to AuditEvent record", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Add SourceNode to SiteCallOperational + SiteCall entity", "status": "pending", "blockedBy": [2]}, {"id": 4, "subject": "Task 3: Add SourceNode to SiteCallOperational + SiteCall entity", "status": "completed", "blockedBy": [2]},
{"id": 5, "subject": "Task 4: Add SourceNode to Notification entity + NotificationSubmit", "status": "pending", "blockedBy": [2]}, {"id": 5, "subject": "Task 4: Add SourceNode to Notification entity + NotificationSubmit", "status": "completed", "blockedBy": [2]},
{"id": 6, "subject": "Task 5: Add source_node to proto + update DTO mappers", "status": "pending", "blockedBy": [3, 4]}, {"id": 6, "subject": "Task 5: Add source_node to proto + update DTO mappers", "status": "completed", "blockedBy": [3, 4]},
{"id": 7, "subject": "Task 6: EF migration — SourceNode on AuditLog + IX_AuditLog_Node_Occurred", "status": "pending", "blockedBy": [3]}, {"id": 7, "subject": "Task 6: EF migration — SourceNode on AuditLog + IX_AuditLog_Node_Occurred", "status": "completed", "blockedBy": [3]},
{"id": 8, "subject": "Task 7: EF migration — SourceNode on Notifications", "status": "pending", "blockedBy": [5]}, {"id": 8, "subject": "Task 7: EF migration — SourceNode on Notifications", "status": "completed", "blockedBy": [5]},
{"id": 9, "subject": "Task 8: EF migration — SourceNode on SiteCalls", "status": "pending", "blockedBy": [4]}, {"id": 9, "subject": "Task 8: EF migration — SourceNode on SiteCalls", "status": "completed", "blockedBy": [4]},
{"id": 10, "subject": "Task 9: Site SQLite AuditLog — add SourceNode (idempotent upgrade)", "status": "pending", "blockedBy": [3]}, {"id": 10, "subject": "Task 9: Site SQLite AuditLog — add SourceNode (idempotent upgrade)", "status": "completed", "blockedBy": [3]},
{"id": 11, "subject": "Task 10: Site SQLite OperationTracking — add SourceNode", "status": "pending", "blockedBy": [4]}, {"id": 11, "subject": "Task 10: Site SQLite OperationTracking — add SourceNode", "status": "completed", "blockedBy": [4]},
{"id": 12, "subject": "Task 11: Stamp SourceNode at site SqliteAuditWriter", "status": "pending", "blockedBy": [2, 10]}, {"id": 12, "subject": "Task 11: Stamp SourceNode at site SqliteAuditWriter", "status": "completed", "blockedBy": [2, 10]},
{"id": 13, "subject": "Task 12: Stamp SourceNode at CentralAuditWriter + persist via repo", "status": "pending", "blockedBy": [2, 7]}, {"id": 13, "subject": "Task 12: Stamp SourceNode at CentralAuditWriter + persist via repo", "status": "completed", "blockedBy": [2, 7]},
{"id": 14, "subject": "Task 13: Carry SourceNode through Notifications S&F handoff", "status": "pending", "blockedBy": [2, 5, 8]}, {"id": 14, "subject": "Task 13: Carry SourceNode through Notifications S&F handoff", "status": "completed", "blockedBy": [2, 5, 8]},
{"id": 15, "subject": "Task 14: Carry SourceNode through cached-call telemetry → SiteCalls", "status": "pending", "blockedBy": [2, 9, 11]}, {"id": 15, "subject": "Task 14: Carry SourceNode through cached-call telemetry → SiteCalls", "status": "completed", "blockedBy": [2, 9, 11]},
{"id": 16, "subject": "Task 15: UI — Node column + filter on AuditLog grid", "status": "pending", "blockedBy": [7]}, {"id": 16, "subject": "Task 15: UI — Node column + filter on AuditLog grid", "status": "completed", "blockedBy": [7]},
{"id": 17, "subject": "Task 16: UI — Node column + filter on Notifications grid", "status": "pending", "blockedBy": [7]}, {"id": 17, "subject": "Task 16: UI — Node column + filter on Notifications grid", "status": "completed", "blockedBy": [7]},
{"id": 18, "subject": "Task 17: UI — Node column + filter on SiteCalls grid", "status": "pending", "blockedBy": [7]}, {"id": 18, "subject": "Task 17: UI — Node column + filter on SiteCalls grid", "status": "completed", "blockedBy": [7]},
{"id": 19, "subject": "Task 18: Docker appsettings — NodeName on all 8 nodes", "status": "pending", "blockedBy": [2]}, {"id": 19, "subject": "Task 18: Docker appsettings — NodeName on all 8 nodes", "status": "completed", "blockedBy": [2]},
{"id": 20, "subject": "Task 19: Full build + targeted test sweep", "status": "pending", "blockedBy": [12, 13, 14, 15, 16, 17, 18, 19]}, {"id": 20, "subject": "Task 19: Full build + targeted test sweep", "status": "completed", "blockedBy": [12, 13, 14, 15, 16, 17, 18, 19]},
{"id": 21, "subject": "Task 20: Docker redeploy + smoke verify", "status": "pending", "blockedBy": [20]} {"id": 21, "subject": "Task 20: Docker redeploy + smoke verify", "status": "completed", "blockedBy": [20]}
], ],
"lastUpdated": "2026-05-23T00:00:00Z" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-23-inbound-api-full-response-audit-design.md
+7 -7
View File
@@ -132,17 +132,17 @@ same "options validation" path used for other AuditLog settings.
## Acceptance Criteria ## Acceptance Criteria
- [ ] `AuditLog:InboundMaxBytes` option exists on the AuditLog options class, - [x] `AuditLog:InboundMaxBytes` option exists on the AuditLog options class,
with the documented default and bounds, validated at startup. with the documented default and bounds, validated at startup.
- [ ] Inbound request middleware writes `RequestSummary` and `ResponseSummary` - [x] Inbound request middleware writes `RequestSummary` and `ResponseSummary`
using the inbound ceiling instead of the 8 KB / 64 KB defaults. using the inbound ceiling instead of the 8 KB / 64 KB defaults.
- [ ] Other channels' rows (e.g. an `ApiOutbound.ApiCall` over the limit) still - [x] Other channels' rows (e.g. an `ApiOutbound.ApiCall` over the limit) still
truncate at 8 KB (64 KB on error rows) — regression-tested. truncate at 8 KB (64 KB on error rows) — regression-tested.
- [ ] `PayloadTruncated = 1` on an inbound row iff request body or response - [x] `PayloadTruncated = 1` on an inbound row iff request body or response
body exceeded `InboundMaxBytes`. body exceeded `InboundMaxBytes`.
- [ ] Header redaction list and per-target body redactors still apply to - [x] Header redaction list and per-target body redactors still apply to
inbound rows. inbound rows.
- [ ] Redactor failure on an inbound row still produces `<redacted: redactor - [x] Redactor failure on an inbound row still produces `<redacted: redactor
error>` and increments `AuditRedactionFailure`. error>` and increments `AuditRedactionFailure`.
- [ ] `Component-AuditLog.md` and `Component-InboundAPI.md` updated as - [x] `Component-AuditLog.md` and `Component-InboundAPI.md` updated as
described in **Doc Edits**. described in **Doc Edits**.
docs/plans/2026-05-23-inbound-api-full-response-audit.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{ {
"planPath": "docs/plans/2026-05-23-inbound-api-full-response-audit.md", "planPath": "docs/plans/2026-05-23-inbound-api-full-response-audit.md",
"tasks": [ "tasks": [
{"id": 1, "subject": "Task 0: Prep — branch, baseline build", "status": "pending"}, {"id": 1, "subject": "Task 0: Prep — branch, baseline build", "status": "completed"},
{"id": 2, "subject": "Task 1: Add InboundMaxBytes to AuditLogOptions (TDD)", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "Task 1: Add InboundMaxBytes to AuditLogOptions (TDD)", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Wire InboundMaxBytes into DefaultAuditPayloadFilter (TDD)", "status": "pending", "blockedBy": [2]}, {"id": 3, "subject": "Task 2: Wire InboundMaxBytes into DefaultAuditPayloadFilter (TDD)", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Capture response body in AuditWriteMiddleware (TDD)", "status": "pending", "blockedBy": [3]}, {"id": 4, "subject": "Task 3: Capture response body in AuditWriteMiddleware (TDD)", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 4: Update Component-AuditLog.md", "status": "pending", "blockedBy": [4]}, {"id": 5, "subject": "Task 4: Update Component-AuditLog.md", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "Task 5: Update Component-InboundAPI.md", "status": "pending", "blockedBy": [5]}, {"id": 6, "subject": "Task 5: Update Component-InboundAPI.md", "status": "completed", "blockedBy": [5]},
{"id": 7, "subject": "Task 6: Final build + full test run + branch summary", "status": "pending", "blockedBy": [6]} {"id": 7, "subject": "Task 6: Final build + full test run + branch summary", "status": "completed", "blockedBy": [6]}
], ],
"lastUpdated": "2026-05-23" "lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-05-24-second-environment-verification.md
+16
View File
@@ -1,5 +1,21 @@
# Env2 + Transport Manual Verification Checklist # Env2 + Transport Manual Verification Checklist
> **RETIRED 2026-08-01 — never run as written; superseded by automated suites + a live cross-environment run.**
> The step list below predates the M8 import-wizard Map step and was never annotated. Instead of
> refreshing ~48 stale steps, the core scenario this checklist exists for was run live on 2026-08-01
> (rig session, task #12): CLI `bundle export` from the primary cluster (encrypted, `--include-dependencies`,
> template closure + shared scripts + external system + site-a + instances) → `bundle preview` against the
> empty env2 (correct Map requirements surfaced) → `bundle import --create-missing-sites
> --create-missing-connections` into env2 → 20 entities added, secrets (`AuthConfiguration`) intact,
> instances landed `NotDeployed`, connection FK wired to the created site. **The run caught a real bug**
> (create-missing site id never materialised before the connection insert → FK 547 on real SQL Server,
> masked by the in-memory test provider) — fixed the same day with an SQLite regression test
> (`CreateMissingSiteRelationalTests`, commit `0c9dffed`). Env2 boot/PSK/LocalDb were separately proven
> PASS 3/3 by Gitea #31 (2026-07-23). Remaining coverage lives in
> `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests` (RoundTrip/RoundTripEquivalence/
> ConflictResolution/ValidationFailure/SiteInstanceImport/CreateMissingSiteRelational). This checklist
> is retired — do not run or refresh it.
**Date created:** 2026-05-24 **Date created:** 2026-05-24
**Companion to:** [`2026-05-24-second-environment-design.md`](2026-05-24-second-environment-design.md), [`Component-Transport.md`](../requirements/Component-Transport.md) **Companion to:** [`2026-05-24-second-environment-design.md`](2026-05-24-second-environment-design.md), [`Component-Transport.md`](../requirements/Component-Transport.md)
**Goal:** Exercise the Transport (#24) bundle export/import flow against two real running environments (primary + env2). **Goal:** Exercise the Transport (#24) bundle export/import flow against two real running environments (primary + env2).
docs/plans/2026-05-24-transport-manual-verification.md
+7
View File
@@ -1,5 +1,12 @@
# Transport Manual Verification # Transport Manual Verification
> **RETIRED 2026-08-01 — never run as written; ~80% overlap with
> `2026-05-24-second-environment-verification.md`, which carries the full retirement rationale.**
> The steps predate the M8 import-wizard Map step (numbering stale). The export→preview→import flow
> was proven live cross-environment on 2026-08-01 via the CLI (see the retirement banner in the env2
> checklist), and the UI wizard surface is covered by the Transport Playwright/bUnit suites. Retired —
> do not run or refresh.
This document is a one-time manual verification to be run against the docker cluster after the Transport feature is fully built and `bash docker/deploy.sh` has rebuilt the image. Each step is sequential and assumes the previous step succeeded. The entire flow takes approximately 15 minutes. This document is a one-time manual verification to be run against the docker cluster after the Transport feature is fully built and `bash docker/deploy.sh` has rebuilt the image. Each step is sequential and assumes the previous step succeeded. The entire flow takes approximately 15 minutes.
## Prerequisites ## Prerequisites
docs/plans/2026-05-28-opcua-tag-browser.md
+10
View File
@@ -1758,6 +1758,16 @@ git commit -m "feat(centralui): add OPC UA browse button + override column to In
### Task 19: End-to-end smoke (manual) ### Task 19: End-to-end smoke (manual)
> **PASS 2026-08-01 (online + offline)** — run live on the docker rig against a purpose-built
> deployed instance (data-sourced Float attribute bound to site-a's "OPC PLC Simulator";
> artifacts deleted after). Online: Browse dialog opened, root populated (Server visible),
> expanded OpcPlc → Telemetry → Basic, selected `StepUp` — footer + override stored the durable
> `nsu=http://microsoft.com/Opc/OpcPlc/;s=StepUp` form and it persisted across save + reload.
> Offline: with BOTH site-a nodes stopped, Browse showed the error banner
> (`StatusCode="Unavailable"` + Retry) with the manual-paste field still usable; a hand-typed
> `ns=3;s=TestChildObject.TestInt` (bare-index advisory shown as designed) went Use → Select →
> Save and persisted across reload while the site was down. Pair restarted staggered and rejoined.
**Classification:** trivial **Classification:** trivial
**Estimated implement time:** ~5 min (manual) **Estimated implement time:** ~5 min (manual)
**Parallelizable with:** none (validates Tasks 1–18 together) **Parallelizable with:** none (validates Tasks 1–18 together)
docs/plans/2026-05-28-opcua-tag-browser.md.tasks.json
+24 -23
View File
@@ -1,28 +1,29 @@
{ {
"planPath": "docs/plans/2026-05-28-opcua-tag-browser.md", "planPath": "docs/plans/2026-05-28-opcua-tag-browser.md",
"tasks": [ "tasks": [
{"id": 70, "subject": "Task 1: Add DataSourceReferenceOverride to InstanceConnectionBinding entity", "status": "pending"}, {"id": 70, "subject": "Task 1: Add DataSourceReferenceOverride to InstanceConnectionBinding entity", "status": "completed"},
{"id": 71, "subject": "Task 2: Add override to ConnectionBinding wire record + ManagementActor mapping", "status": "pending"}, {"id": 71, "subject": "Task 2: Add override to ConnectionBinding wire record + ManagementActor mapping", "status": "completed"},
{"id": 72, "subject": "Task 3: EF mapping for DataSourceReferenceOverride column", "status": "pending"}, {"id": 72, "subject": "Task 3: EF mapping for DataSourceReferenceOverride column", "status": "completed"},
{"id": 73, "subject": "Task 4: EF Core migration AddInstanceConnectionBindingOverride", "status": "pending", "blockedBy": [72]}, {"id": 73, "subject": "Task 4: EF Core migration AddInstanceConnectionBindingOverride", "status": "completed", "blockedBy": [72]},
{"id": 74, "subject": "Task 5: IBrowsableDataConnection interface + BrowseNode types", "status": "pending"}, {"id": 74, "subject": "Task 5: IBrowsableDataConnection interface + BrowseNode types", "status": "completed"},
{"id": 75, "subject": "Task 6: BrowseCommands.cs (BrowseOpcUaNodeCommand + result + failure)", "status": "pending"}, {"id": 75, "subject": "Task 6: BrowseCommands.cs (BrowseOpcUaNodeCommand + result + failure)", "status": "completed"},
{"id": 76, "subject": "Task 7: Add BrowseChildrenAsync to IOpcUaClient", "status": "pending", "blockedBy": [74, 75]}, {"id": 76, "subject": "Task 7: Add BrowseChildrenAsync to IOpcUaClient", "status": "completed", "blockedBy": [74, 75]},
{"id": 77, "subject": "Task 8: Implement BrowseChildrenAsync on RealOpcUaClient", "status": "pending", "blockedBy": [76]}, {"id": 77, "subject": "Task 8: Implement BrowseChildrenAsync on RealOpcUaClient", "status": "completed", "blockedBy": [76]},
{"id": 78, "subject": "Task 9: Implement IBrowsableDataConnection on OpcUaDataConnection", "status": "pending", "blockedBy": [76]}, {"id": 78, "subject": "Task 9: Implement IBrowsableDataConnection on OpcUaDataConnection", "status": "completed", "blockedBy": [76]},
{"id": 79, "subject": "Task 10: Handle BrowseOpcUaNodeCommand in DataConnectionManagerActor", "status": "pending", "blockedBy": [75, 76]}, {"id": 79, "subject": "Task 10: Handle BrowseOpcUaNodeCommand in DataConnectionManagerActor", "status": "completed", "blockedBy": [75, 76]},
{"id": 80, "subject": "Task 11: Forward BrowseOpcUaNodeCommand in SiteCommunicationActor", "status": "pending", "blockedBy": [79]}, {"id": 80, "subject": "Task 11: Forward BrowseOpcUaNodeCommand in SiteCommunicationActor", "status": "completed", "blockedBy": [79]},
{"id": 81, "subject": "Task 12: Apply override in FlatteningService.ApplyConnectionBindings", "status": "pending", "blockedBy": [70]}, {"id": 81, "subject": "Task 12: Apply override in FlatteningService.ApplyConnectionBindings", "status": "completed", "blockedBy": [70]},
{"id": 82, "subject": "Task 13: Revision-hash regression test", "status": "pending", "blockedBy": [81]}, {"id": 82, "subject": "Task 13: Revision-hash regression test", "status": "completed", "blockedBy": [81]},
{"id": 83, "subject": "Task 14: IOpcUaBrowseService + impl + DI registration", "status": "pending", "blockedBy": [75]}, {"id": 83, "subject": "Task 14: IOpcUaBrowseService + impl + DI registration", "status": "completed", "blockedBy": [75]},
{"id": 84, "subject": "Task 15: Scaffold OpcUaBrowserDialog modal", "status": "pending"}, {"id": 84, "subject": "Task 15: Scaffold OpcUaBrowserDialog modal", "status": "completed"},
{"id": 85, "subject": "Task 16: Tree rendering + lazy load + selection in dialog", "status": "pending", "blockedBy": [83, 84]}, {"id": 85, "subject": "Task 16: Tree rendering + lazy load + selection in dialog", "status": "completed", "blockedBy": [83, 84]},
{"id": 86, "subject": "Task 17: Error banner mapping + polish on dialog", "status": "pending", "blockedBy": [85]}, {"id": 86, "subject": "Task 17: Error banner mapping + polish on dialog", "status": "completed", "blockedBy": [85]},
{"id": 87, "subject": "Task 18: Add Override column + Browse button to InstanceConfigure.razor", "status": "pending", "blockedBy": [83, 86]}, {"id": 87, "subject": "Task 18: Add Override column + Browse button to InstanceConfigure.razor", "status": "completed", "blockedBy": [83, 86]},
{"id": 88, "subject": "Task 19: End-to-end manual smoke (online + offline)", "status": "pending", "blockedBy": [70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87]}, {"id": 88, "subject": "Task 19: End-to-end manual smoke (online + offline)", "status": "completed", "blockedBy": [70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87], "notes": "Completed 2026-08-01: manual smoke PASS, online + offline paths verified live (commit 6dc5d94c). Status flipped from stale 'pending' by the 2026-08-07 truth sweep."},
{"id": 89, "subject": "Task 20: Update Component-DataConnectionLayer.md", "status": "pending", "blockedBy": [81]}, {"id": 89, "subject": "Task 20: Update Component-DataConnectionLayer.md", "status": "completed", "blockedBy": [81]},
{"id": 90, "subject": "Task 21: Update Component-TemplateEngine.md", "status": "pending", "blockedBy": [81]}, {"id": 90, "subject": "Task 21: Update Component-TemplateEngine.md", "status": "completed", "blockedBy": [81]},
{"id": 91, "subject": "Task 22: Update Component-CentralUI.md", "status": "pending", "blockedBy": [87]} {"id": 91, "subject": "Task 22: Update Component-CentralUI.md", "status": "completed", "blockedBy": [87]}
], ],
"lastUpdated": "2026-05-28T00:00:00Z" "status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
} }
docs/plans/2026-05-28-scadabridge-rename-design.md
+2 -2
View File
@@ -1,7 +1,7 @@
# ScadaLink → ZB.MOM.WW.ScadaBridge Rename — Design # ScadaLink → ZB.MOM.WW.ScadaBridge Rename — Design
**Date:** 2026-05-28 **Date:** 2026-05-28
**Status:** Approved, in implementation **Status:** Implemented (commit `7b0b9c73`)
**Scope:** Repo-wide rename of the product from "ScadaLink" to "ScadaBridge" and addition of the `ZB.MOM.WW` company prefix to every .NET project. Code, runtime artifacts (containers, network, databases), docs, and CLI config. **Scope:** Repo-wide rename of the product from "ScadaLink" to "ScadaBridge" and addition of the `ZB.MOM.WW` company prefix to every .NET project. Code, runtime artifacts (containers, network, databases), docs, and CLI config.
## Decisions ## Decisions
@@ -115,7 +115,7 @@ Drop volume + re-seed. `docker compose -f infra/docker-compose.yml down -v` then
## Out of Scope ## Out of Scope
- Sister repos `~/Desktop/MxAccessGateway`, `~/Desktop/OtOpcUa` — independent codebases, independent timelines. - Sister repos `~/Desktop/MxAccessGateway`, `~/Desktop/OtOpcUa` — independent codebases, independent timelines.
- Repo folder `~/Desktop/scadalink-design` — left as-is to preserve Claude Code memory paths and user shell context. - Repo folder `~/Desktop/scadalink-design` — left as-is to preserve Claude Code memory paths and user shell context. *(2026-08-01: this exclusion was later reversed and completed — see `2026-05-31-folder-repo-rename-scadabridge-plan.md` / `-design.md`, which renamed the folder and the Gitea repo to ScadaBridge.)*
- Existing Transport bundle manifests with `SourceEnvironment = "docker-cluster"` — those values are deployment IDs and stay stable. If the user later wants to rename the deployment IDs themselves, that's a separate design. - Existing Transport bundle manifests with `SourceEnvironment = "docker-cluster"` — those values are deployment IDs and stay stable. If the user later wants to rename the deployment IDs themselves, that's a separate design.
## Verification ## Verification
docs/plans/2026-06-03-component-reference-docs.md.tasks.json
+30 -29
View File
@@ -1,34 +1,35 @@
{ {
"planPath": "docs/plans/2026-06-03-component-reference-docs.md", "planPath": "docs/plans/2026-06-03-component-reference-docs.md",
"tasks": [ "tasks": [
{"id": 19, "subject": "Task 0: Scaffold + shared assets", "status": "pending"}, {"id": 19, "subject": "Task 0: Scaffold + shared assets", "status": "completed"},
{"id": 20, "subject": "Task 1: Pilot exemplar AuditLog.md (approval gate)", "status": "pending", "blockedBy": [19]}, {"id": 20, "subject": "Task 1: Pilot exemplar AuditLog.md (approval gate)", "status": "completed", "blockedBy": [19]},
{"id": 21, "subject": "Task 2: Commons.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 21, "subject": "Task 2: Commons.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 3: ConfigurationDatabase.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 22, "subject": "Task 3: ConfigurationDatabase.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 23, "subject": "Task 4: Communication.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 23, "subject": "Task 4: Communication.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 24, "subject": "Task 5: ClusterInfrastructure.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 24, "subject": "Task 5: ClusterInfrastructure.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 25, "subject": "Task 6: Host.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 25, "subject": "Task 6: Host.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 26, "subject": "Task 7: Security.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 26, "subject": "Task 7: Security.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 27, "subject": "Task 8: TemplateEngine.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 27, "subject": "Task 8: TemplateEngine.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 28, "subject": "Task 9: DeploymentManager.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 28, "subject": "Task 9: DeploymentManager.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 29, "subject": "Task 10: SiteRuntime.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 29, "subject": "Task 10: SiteRuntime.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 30, "subject": "Task 11: DataConnectionLayer.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 30, "subject": "Task 11: DataConnectionLayer.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 31, "subject": "Task 12: StoreAndForward.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 31, "subject": "Task 12: StoreAndForward.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 32, "subject": "Task 13: ExternalSystemGateway.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 32, "subject": "Task 13: ExternalSystemGateway.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 33, "subject": "Task 14: NotificationService.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 33, "subject": "Task 14: NotificationService.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 34, "subject": "Task 15: NotificationOutbox.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 34, "subject": "Task 15: NotificationOutbox.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 35, "subject": "Task 16: SiteCallAudit.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 35, "subject": "Task 16: SiteCallAudit.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 36, "subject": "Task 17: HealthMonitoring.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 36, "subject": "Task 17: HealthMonitoring.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 37, "subject": "Task 18: SiteEventLogging.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 37, "subject": "Task 18: SiteEventLogging.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 38, "subject": "Task 19: InboundAPI.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 38, "subject": "Task 19: InboundAPI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 39, "subject": "Task 20: ManagementService.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 39, "subject": "Task 20: ManagementService.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 40, "subject": "Task 21: CLI.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 40, "subject": "Task 21: CLI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 41, "subject": "Task 22: Transport.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 41, "subject": "Task 22: Transport.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 42, "subject": "Task 23: CentralUI.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 42, "subject": "Task 23: CentralUI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 43, "subject": "Task 24: TraefikProxy.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 43, "subject": "Task 24: TraefikProxy.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 44, "subject": "Task 25: TreeView.md reference doc", "status": "pending", "blockedBy": [20]}, {"id": 44, "subject": "Task 25: TreeView.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 45, "subject": "Task 26: Index + README link", "status": "pending", "blockedBy": [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]}, {"id": 45, "subject": "Task 26: Index + README link", "status": "completed", "blockedBy": [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]},
{"id": 46, "subject": "Task 27: Verification & fix pass", "status": "pending", "blockedBy": [45]} {"id": 46, "subject": "Task 27: Verification & fix pass", "status": "completed", "blockedBy": [45]}
], ],
"lastUpdated": "2026-06-03" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-05-playwright-coverage-expansion.md.tasks.json
+18 -17
View File
@@ -1,23 +1,24 @@
{ {
"planPath": "docs/plans/2026-06-05-playwright-coverage-expansion.md", "planPath": "docs/plans/2026-06-05-playwright-coverage-expansion.md",
"lastUpdated": "2026-06-05T00:00:00Z", "lastUpdated": "2026-08-01T00:00:00Z",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"nativeTaskIdBase": 57, "nativeTaskIdBase": 57,
"tasks": [ "tasks": [
{"id": 0, "nativeId": 57, "subject": "Task 0: Add CLI ProjectReference to test project", "status": "pending"}, {"id": 0, "nativeId": 57, "subject": "Task 0: Add CLI ProjectReference to test project", "status": "completed"},
{"id": 1, "nativeId": 58, "subject": "Task 1: CliRunner core + ClusterAvailability probe", "status": "pending", "blockedBy": [0]}, {"id": 1, "nativeId": 58, "subject": "Task 1: CliRunner core + ClusterAvailability probe", "status": "completed", "blockedBy": [0]},
{"id": 2, "nativeId": 59, "subject": "Task 2: CliRunner typed fixture helpers", "status": "pending", "blockedBy": [1]}, {"id": 2, "nativeId": 59, "subject": "Task 2: CliRunner typed fixture helpers", "status": "completed", "blockedBy": [1]},
{"id": 3, "nativeId": 60, "subject": "Task 3: Standardize skip policy + skip-count logging", "status": "pending", "blockedBy": [1]}, {"id": 3, "nativeId": 60, "subject": "Task 3: Standardize skip policy + skip-count logging", "status": "completed", "blockedBy": [1]},
{"id": 4, "nativeId": 61, "subject": "Task 4: DeploymentFixture (ephemeral instance on site-a)", "status": "pending", "blockedBy": [2]}, {"id": 4, "nativeId": 61, "subject": "Task 4: DeploymentFixture (ephemeral instance on site-a)", "status": "completed", "blockedBy": [2]},
{"id": 5, "nativeId": 62, "subject": "Task 5: DeploymentActionTests.Deploy", "status": "pending", "blockedBy": [4]}, {"id": 5, "nativeId": 62, "subject": "Task 5: DeploymentActionTests.Deploy", "status": "completed", "blockedBy": [4]},
{"id": 6, "nativeId": 63, "subject": "Task 6: DeploymentActionTests.Enable + Disable", "status": "pending", "blockedBy": [5]}, {"id": 6, "nativeId": 63, "subject": "Task 6: DeploymentActionTests.Enable + Disable", "status": "completed", "blockedBy": [5]},
{"id": 7, "nativeId": 64, "subject": "Task 7: DeploymentActionTests.Delete", "status": "pending", "blockedBy": [6]}, {"id": 7, "nativeId": 64, "subject": "Task 7: DeploymentActionTests.Delete", "status": "completed", "blockedBy": [6]},
{"id": 8, "nativeId": 65, "subject": "Task 8: Notification retry/discard + ParkedMessages query", "status": "pending", "blockedBy": [2]}, {"id": 8, "nativeId": 65, "subject": "Task 8: Notification retry/discard + ParkedMessages query", "status": "completed", "blockedBy": [2]},
{"id": 9, "nativeId": 66, "subject": "Task 9: Transport Import round-trip", "status": "pending", "blockedBy": [2]}, {"id": 9, "nativeId": 66, "subject": "Task 9: Transport Import round-trip", "status": "completed", "blockedBy": [2]},
{"id": 10, "nativeId": 67, "subject": "Task 10: Site CRUD round-trip", "status": "pending", "blockedBy": [2]}, {"id": 10, "nativeId": 67, "subject": "Task 10: Site CRUD round-trip", "status": "completed", "blockedBy": [2]},
{"id": 11, "nativeId": 68, "subject": "Task 11: Template CRUD round-trip", "status": "pending", "blockedBy": [2]}, {"id": 11, "nativeId": 68, "subject": "Task 11: Template CRUD round-trip", "status": "completed", "blockedBy": [2]},
{"id": 12, "nativeId": 69, "subject": "Task 12: LDAP mapping CRUD round-trip", "status": "pending", "blockedBy": [1]}, {"id": 12, "nativeId": 69, "subject": "Task 12: LDAP mapping CRUD round-trip", "status": "completed", "blockedBy": [1]},
{"id": 13, "nativeId": 70, "subject": "Task 13: Navigation render-assertion hardening", "status": "pending"}, {"id": 13, "nativeId": 70, "subject": "Task 13: Navigation render-assertion hardening", "status": "completed"},
{"id": 14, "nativeId": 71, "subject": "Task 14: Health KPI load test", "status": "pending", "blockedBy": [1]}, {"id": 14, "nativeId": 71, "subject": "Task 14: Health KPI load test", "status": "completed", "blockedBy": [1]},
{"id": 15, "nativeId": 72, "subject": "Task 15: Full-suite verification + no-residue check", "status": "pending", "blockedBy": [3, 7, 8, 9, 10, 11, 12, 13, 14]} {"id": 15, "nativeId": 72, "subject": "Task 15: Full-suite verification + no-residue check", "status": "completed", "blockedBy": [3, 7, 8, 9, 10, 11, 12, 13, 14]}
] ]
} }
docs/plans/2026-06-15-stillpending-completion-design.md
+5 -2
View File
@@ -78,7 +78,10 @@ Wire up behavior that exists in code but is never started, and fill the event-lo
#### M5 — Audit hardening (T1–T8) #### M5 — Audit hardening (T1–T8)
Hash-chain tamper evidence (off by default, `verify-chain` made real); Parquet export/archival (replace the 501); per-channel retention overrides; tag-cascade for `ParentExecutionId` (thread writing-execution id through trigger-driven runs); ExecutionId/ParentExecutionId + SourceNode backfill on historical rows; per-node stuck-count KPIs; structured response capture (headers/content-type, inbound request headers, per-method opt-out, `AuditInboundCeilingHits` metric); CLI `audit tree`. Hash-chain tamper evidence (off by default, `verify-chain` made real); Parquet export/archival (replace the 501); per-channel retention overrides; tag-cascade for `ParentExecutionId` (thread writing-execution id through trigger-driven runs); ExecutionId/ParentExecutionId + SourceNode backfill on historical rows; per-node stuck-count KPIs; structured response capture (headers/content-type, inbound request headers, per-method opt-out, `AuditInboundCeilingHits` metric); CLI `audit tree`.
#### M6 — KPI History & Trends (T11 delivered; T9/T10 deferred) #### M6 — KPI History & Trends (T11 delivered; T9/T10 **since DELIVERED 2026-06-19**)
> **2026-08-01 bookkeeping sync:** the T9/T10 "deferred to the next major version" text below is superseded — **T9/T10 were delivered 2026-06-19 as the SMS (Twilio) adapter** (`SmsNotificationDeliveryAdapter`, `NotificationType.Sms`, Central UI Type selector + SMS recipient input; see `docs/plans/2026-06-19-sms-notifications.md`). **Teams was evaluated and dropped by design decision** — outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only.
Reshaped during the 2026-06-17 brainstorm (see `docs/plans/2026-06-17-m6-kpi-history-design.md`): Reshaped during the 2026-06-17 brainstorm (see `docs/plans/2026-06-17-m6-kpi-history-design.md`):
- **T11 — DELIVERED** as the reusable **KPI-history backbone** (#26 KpiHistory), promoted from a notifications-only feature. A tall/EAV `KpiSample` store in **central MS SQL** (no new infra — supersedes the original "point-in-time only, no time-series store" stance), a `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated, best-effort with per-source isolation) sampling DI-registered `IKpiSampleSource`s every minute, a bucketed `GetRawSeriesAsync` + `KpiSeriesBucketer` query + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart` (no third-party charting lib). Trends shipped for **all** current KPI sources — Notification Outbox, Site Call Audit, Audit Log, and Site Health — across four UI surfaces. - **T11 — DELIVERED** as the reusable **KPI-history backbone** (#26 KpiHistory), promoted from a notifications-only feature. A tall/EAV `KpiSample` store in **central MS SQL** (no new infra — supersedes the original "point-in-time only, no time-series store" stance), a `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated, best-effort with per-source isolation) sampling DI-registered `IKpiSampleSource`s every minute, a bucketed `GetRawSeriesAsync` + `KpiSeriesBucketer` query + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart` (no third-party charting lib). Trends shipped for **all** current KPI sources — Notification Outbox, Site Call Audit, Audit Log, and Site Health — across four UI surfaces.
- **T9 (Teams + other non-Email delivery adapters behind `INotificationDeliveryAdapter`) — DEFERRED to the next major version.** The seam exists; no code now. Transport choice (Incoming Webhook vs Microsoft Graph) and the Teams list-targeting model remain to be designed. - **T9 (Teams + other non-Email delivery adapters behind `INotificationDeliveryAdapter`) — DEFERRED to the next major version.** The seam exists; no code now. Transport choice (Incoming Webhook vs Microsoft Graph) and the Teams list-targeting model remain to be designed.
@@ -132,7 +135,7 @@ This is the **final milestone** of the system-completion roadmap. All in-scope M
## Dependencies & sequencing ## Dependencies & sequencing
- **M1 → M5** — audit hardening builds on the wired purge/reconciliation. - **M1 → M5** — audit hardening builds on the wired purge/reconciliation.
- **M6/T11** — delivered as the #26 KpiHistory backbone; reused **central MS SQL** (a tall/EAV `KpiSample` table) rather than introducing new infra. T9/T10 deferred to the next major version. - **M6/T11** — delivered as the #26 KpiHistory backbone; reused **central MS SQL** (a tall/EAV `KpiSample` table) rather than introducing new infra. T9/T10 since delivered 2026-06-19 as the SMS (Twilio) adapter (see note above).
- **M9/T26** — base-template versioning is the largest authoring item; may split. - **M9/T26** — base-template versioning is the largest authoring item; may split.
- **M4** — runs anytime; cheap and high-clarity, good to interleave. - **M4** — runs anytime; cheap and high-clarity, good to interleave.
- **M3** — independent; can run in parallel with M1/M2. - **M3** — independent; can run in parallel with M1/M2.
docs/plans/2026-06-15-stillpending-m2-implementation.md.tasks.json
+5 -4
View File
@@ -27,9 +27,10 @@
{"ref": "#19", "subject": "script started/completed events", "status": "done in M1.8"} {"ref": "#19", "subject": "script started/completed events", "status": "done in M1.8"}
], ],
"followups": [ "followups": [
{"id": 52, "subject": "Investigate 2 partition-purge E2E test failures (AuditLogPurgeActor/PartitionPurge)", "from": "M2.0", "status": "pending"}, {"id": 52, "subject": "Investigate 2 partition-purge E2E test failures (AuditLogPurgeActor/PartitionPurge)", "from": "M2.0", "status": "resolved", "note": "RESOLVED — commit 639e331d de-dated the 2 EndToEnd purge tests (M5.7)"},
{"id": 53, "subject": "Dedup alarm-capable protocol predicate (3 copies → AlarmCapableProtocols)", "from": "M2.1", "status": "pending"}, {"id": 53, "subject": "Dedup alarm-capable protocol predicate (3 copies → AlarmCapableProtocols)", "from": "M2.1", "status": "resolved", "note": "RESOLVED — shared Commons/Interfaces/Protocol/AlarmCapableProtocols.cs is now the single predicate"},
{"id": 54, "subject": "Expose ExecutionTimeoutSeconds (+ MinTimeBetweenRuns) in CLI + UI script authoring", "from": "M2.5", "status": "pending"} {"id": 54, "subject": "Expose ExecutionTimeoutSeconds (+ MinTimeBetweenRuns) in CLI + UI script authoring", "from": "M2.5", "status": "resolved", "note": "RESOLVED — CLI --execution-timeout-seconds on template script add/update plus TemplateEdit binding"}
], ],
"lastUpdated": "2026-06-15" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-15-stillpending-phase1-implementation.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{ {
"planPath": "docs/plans/2026-06-15-stillpending-phase1-implementation.md", "planPath": "docs/plans/2026-06-15-stillpending-phase1-implementation.md",
"tasks": [ "tasks": [
{"id": 22, "subject": "M1.0: Confirm proto/site surface for audit pull (spike)", "status": "pending"}, {"id": 22, "subject": "M1.0: Confirm proto/site surface for audit pull (spike)", "status": "completed"},
{"id": 23, "subject": "M1.1: Production gRPC IPullAuditEventsClient", "status": "pending", "blockedBy": [22]}, {"id": 23, "subject": "M1.1: Production gRPC IPullAuditEventsClient", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "M1.2: Wire reconciliation + purge actors as central singletons", "status": "pending"}, {"id": 24, "subject": "M1.2: Wire reconciliation + purge actors as central singletons", "status": "completed"},
{"id": 25, "subject": "M1.3: SiteCallAudit periodic reconciliation pull", "status": "pending"}, {"id": 25, "subject": "M1.3: SiteCallAudit periodic reconciliation pull", "status": "completed"},
{"id": 26, "subject": "M1.4: SiteCallAudit daily terminal-row purge scheduler", "status": "pending"}, {"id": 26, "subject": "M1.4: SiteCallAudit daily terminal-row purge scheduler", "status": "completed"},
{"id": 27, "subject": "M1.5: SiteEventLog — emit Alarm events", "status": "pending"}, {"id": 27, "subject": "M1.5: SiteEventLog — emit Alarm events", "status": "completed"},
{"id": 28, "subject": "M1.6: SiteEventLog — Deployment + Instance-lifecycle events", "status": "pending"}, {"id": 28, "subject": "M1.6: SiteEventLog — Deployment + Instance-lifecycle events", "status": "completed"},
{"id": 29, "subject": "M1.7: SiteEventLog — Store-and-Forward + Notification events", "status": "pending"}, {"id": 29, "subject": "M1.7: SiteEventLog — Store-and-Forward + Notification events", "status": "completed"},
{"id": 30, "subject": "M1.8: SiteEventLog — script started/completed (Info)", "status": "pending"}, {"id": 30, "subject": "M1.8: SiteEventLog — script started/completed (Info)", "status": "completed"},
{"id": 31, "subject": "M1.9: M1 integration verification + redeploy", "status": "pending", "blockedBy": [23, 24, 25, 26, 27, 28, 29, 30]}, {"id": 31, "subject": "M1.9: M1 integration verification + redeploy", "status": "completed", "blockedBy": [23, 24, 25, 26, 27, 28, 29, 30]},
{"id": 13, "subject": "M2 — Correctness & behavioral gaps (Tier 2) [umbrella; split per-item at execution]", "status": "pending"}, {"id": 13, "subject": "M2 — Correctness & behavioral gaps (Tier 2) [umbrella; split per-item at execution]", "status": "completed"},
{"id": 14, "subject": "M3 — Script trust boundary (Tier 1 #1-#2) [umbrella]", "status": "pending"}, {"id": 14, "subject": "M3 — Script trust boundary (Tier 1 #1-#2) [umbrella]", "status": "completed"},
{"id": 15, "subject": "M4 — Doc reconciliation (Tier 4) [umbrella]", "status": "pending"} {"id": 15, "subject": "M4 — Doc reconciliation (Tier 4) [umbrella]", "status": "completed"}
], ],
"lastUpdated": "2026-06-15" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-16-ipsen-mes-movein.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{ {
"planPath": "docs/plans/2026-06-16-ipsen-mes-movein.md", "planPath": "docs/plans/2026-06-16-ipsen-mes-movein.md",
"tasks": [ "tasks": [
{"id": 1, "subject": "Task 1: InboundDatabaseHelper (read-only DB for inbound scripts)", "classification": "standard", "status": "pending"}, {"id": 1, "subject": "Task 1: InboundDatabaseHelper (read-only DB for inbound scripts)", "classification": "standard", "status": "completed"},
{"id": 2, "subject": "Task 2: Expose Database to inbound scripts (context + executor)", "classification": "high-risk", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "Task 2: Expose Database to inbound scripts (context + executor)", "classification": "high-risk", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Build + regression-test ScadaBridge", "classification": "small", "status": "pending", "blockedBy": [2]}, {"id": 3, "subject": "Task 3: Build + regression-test ScadaBridge", "classification": "small", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 4: Commit Component A", "classification": "trivial", "status": "pending", "blockedBy": [3]}, {"id": 4, "subject": "Task 4: Commit Component A", "classification": "trivial", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03", "classification": "high-risk", "status": "pending", "blockedBy": [4]}, {"id": 5, "subject": "Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03", "classification": "high-risk", "status": "completed", "note": "done off-repo on wonder-app-vd03 (live methods authored against Database.* fixed there 2026-06-25)", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: Apply Component B - update inbound IpsenMESMoveIn script", "classification": "high-risk", "status": "pending", "blockedBy": [5]}, {"id": 6, "subject": "Task 6: Apply Component B - update inbound IpsenMESMoveIn script", "classification": "high-risk", "status": "completed", "note": "done off-repo on wonder-app-vd03 (live methods authored against Database.* fixed there 2026-06-25)", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Apply Component C - IpsenMoveIn template script on T1", "classification": "high-risk", "status": "pending", "blockedBy": [5]}, {"id": 7, "subject": "Task 7: Apply Component C - IpsenMoveIn template script on T1", "classification": "high-risk", "status": "pending", "note": "NEEDS LIVE vd03 verification — tracked in the 2026-08-01 pending-work task list", "blockedBy": [5]},
{"id": 8, "subject": "Task 8: On-box end-to-end verification", "classification": "standard", "status": "pending", "blockedBy": [6, 7]} {"id": 8, "subject": "Task 8: On-box end-to-end verification", "classification": "standard", "status": "pending", "note": "NEEDS LIVE vd03 verification — tracked in the 2026-08-01 pending-work task list", "blockedBy": [6, 7]}
], ],
"lastUpdated": "2026-06-16" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-16-m5-audit-hardening.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{ {
"planPath": "docs/plans/2026-06-16-m5-audit-hardening.md", "planPath": "docs/plans/2026-06-16-m5-audit-hardening.md",
"tasks": [ "tasks": [
{"id": 119, "subject": "M5.1 (T8): CLI audit tree + tree endpoint", "status": "pending"}, {"id": 119, "subject": "M5.1 (T8): CLI audit tree + tree endpoint", "status": "completed"},
{"id": 120, "subject": "M5.2 (T6): Per-node stuck-count KPIs", "status": "pending"}, {"id": 120, "subject": "M5.2 (T6): Per-node stuck-count KPIs", "status": "completed"},
{"id": 121, "subject": "M5.3 (T7): Structured response-capture increments", "status": "pending"}, {"id": 121, "subject": "M5.3 (T7): Structured response-capture increments", "status": "completed"},
{"id": 122, "subject": "M5.4 (T4): ParentExecutionId tag-cascade", "status": "pending"}, {"id": 122, "subject": "M5.4 (T4): ParentExecutionId tag-cascade", "status": "completed"},
{"id": 123, "subject": "M5.5 (T3): Per-channel retention overrides", "status": "pending"}, {"id": 123, "subject": "M5.5 (T3): Per-channel retention overrides", "status": "completed"},
{"id": 124, "subject": "M5.6 (T5): SourceNode sentinel backfill + runbook", "status": "pending", "blockedBy": [119]}, {"id": 124, "subject": "M5.6 (T5): SourceNode sentinel backfill + runbook", "status": "completed", "blockedBy": [119]},
{"id": 125, "subject": "M5.7: M5 integration verification + docs", "status": "pending", "blockedBy": [119, 120, 121, 122, 123, 124]} {"id": 125, "subject": "M5.7: M5 integration verification + docs", "status": "completed", "blockedBy": [119, 120, 121, 122, 123, 124]}
], ],
"lastUpdated": "2026-06-16" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-16-script-analysis-consolidation.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{ {
"planPath": "docs/plans/2026-06-16-script-analysis-consolidation.md", "planPath": "docs/plans/2026-06-16-script-analysis-consolidation.md",
"tasks": [ "tasks": [
{"id": 103, "subject": "M3.0: Spike — compile-surface stub feasibility + package refs", "status": "pending"}, {"id": 103, "subject": "M3.0: Spike — compile-surface stub feasibility + package refs", "status": "completed"},
{"id": 104, "subject": "M3.1: Build ScriptAnalysis project (policy+validator+compiler+surfaces+tests)", "status": "pending", "blockedBy": [103]}, {"id": 104, "subject": "M3.1: Build ScriptAnalysis project (policy+validator+compiler+surfaces+tests)", "status": "completed", "blockedBy": [103]},
{"id": 105, "subject": "M3.2: TemplateEngine deploy gate → shared analyzer", "status": "pending", "blockedBy": [104]}, {"id": 105, "subject": "M3.2: TemplateEngine deploy gate → shared analyzer", "status": "completed", "blockedBy": [104]},
{"id": 106, "subject": "M3.3: SiteRuntime → shared analyzer + parity test", "status": "pending", "blockedBy": [104]}, {"id": 106, "subject": "M3.3: SiteRuntime → shared analyzer + parity test", "status": "completed", "blockedBy": [104]},
{"id": 107, "subject": "M3.4: InboundAPI → shared analyzer", "status": "pending", "blockedBy": [104]}, {"id": 107, "subject": "M3.4: InboundAPI → shared analyzer", "status": "completed", "blockedBy": [104]},
{"id": 108, "subject": "M3.5: CentralUI → shared analyzer (keep markers + Test-Run host)", "status": "pending", "blockedBy": [104]}, {"id": 108, "subject": "M3.5: CentralUI → shared analyzer (keep markers + Test-Run host)", "status": "completed", "blockedBy": [104]},
{"id": 109, "subject": "M3.6: Integration verification + docs + fixture cleanup", "status": "pending", "blockedBy": [105, 106, 107, 108]} {"id": 109, "subject": "M3.6: Integration verification + docs + fixture cleanup", "status": "completed", "blockedBy": [105, 106, 107, 108]}
], ],
"lastUpdated": "2026-06-16" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-17-debugview-tabs-trees.md.tasks.json
+9 -8
View File
@@ -1,13 +1,13 @@
{ {
"planPath": "docs/plans/2026-06-17-debugview-tabs-trees.md", "planPath": "docs/plans/2026-06-17-debugview-tabs-trees.md",
"tasks": [ "tasks": [
{"id": 170, "subject": "DV-1: Native-binding linkage — additive AlarmStateChanged contract chain", "classification": "high-risk", "status": "pending"}, {"id": 170, "subject": "DV-1: Native-binding linkage — additive AlarmStateChanged contract chain", "classification": "high-risk", "status": "completed"},
{"id": 171, "subject": "DV-2: Site snapshot — placeholder rows for idle native source bindings", "classification": "standard", "status": "pending", "blockedBy": [170]}, {"id": 171, "subject": "DV-2: Site snapshot — placeholder rows for idle native source bindings", "classification": "standard", "status": "completed", "blockedBy": [170]},
{"id": 172, "subject": "DV-3: DebugTreeNode model + attribute-tree builder", "classification": "standard", "status": "pending", "blockedBy": [170]}, {"id": 172, "subject": "DV-3: DebugTreeNode model + attribute-tree builder", "classification": "standard", "status": "completed", "blockedBy": [170]},
{"id": 173, "subject": "DV-4: Alarm-tree builder — computed leaves + native binding grouping + roll-up", "classification": "standard", "status": "pending", "blockedBy": [172, 170]}, {"id": 173, "subject": "DV-4: Alarm-tree builder — computed leaves + native binding grouping + roll-up", "classification": "standard", "status": "completed", "blockedBy": [172, 170]},
{"id": 174, "subject": "DV-5: DebugView page — tabs + two TreeViews + in-place updates", "classification": "standard", "status": "pending", "blockedBy": [173]}, {"id": 174, "subject": "DV-5: DebugView page — tabs + two TreeViews + in-place updates", "classification": "standard", "status": "completed", "blockedBy": [173]},
{"id": 175, "subject": "DV-6: Documentation — CentralUI + SiteRuntime + streaming contract", "classification": "small", "status": "pending", "blockedBy": [173]}, {"id": 175, "subject": "DV-6: Documentation — CentralUI + SiteRuntime + streaming contract", "classification": "small", "status": "completed", "blockedBy": [173]},
{"id": 176, "subject": "DV-7: Integration — full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "pending", "blockedBy": [174, 175, 171]} {"id": 176, "subject": "DV-7: Integration — full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "completed", "blockedBy": [174, 175, 171]}
], ],
"waves": [ "waves": [
{"wave": 1, "tasks": [170]}, {"wave": 1, "tasks": [170]},
@@ -16,5 +16,6 @@
{"wave": 4, "tasks": [174, 175]}, {"wave": 4, "tasks": [174, 175]},
{"wave": 5, "tasks": [176]} {"wave": 5, "tasks": [176]}
], ],
"lastUpdated": "2026-06-17" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-17-m6-kpi-history.md.tasks.json
+19 -18
View File
@@ -1,23 +1,24 @@
{ {
"planPath": "docs/plans/2026-06-17-m6-kpi-history.md", "planPath": "docs/plans/2026-06-17-m6-kpi-history.md",
"tasks": [ "tasks": [
{"id": 1, "subject": "K1: Foundation contracts (Commons)", "nativeId": 136, "classification": "high-risk", "status": "pending"}, {"id": 1, "subject": "K1: Foundation contracts (Commons)", "nativeId": 136, "classification": "high-risk", "status": "completed"},
{"id": 2, "subject": "K2: Persistence — EF config + repository + migration", "nativeId": 137, "classification": "high-risk", "status": "pending", "blockedBy": [1]}, {"id": 2, "subject": "K2: Persistence — EF config + repository + migration", "nativeId": 137, "classification": "high-risk", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "K3: KpiHistory project scaffold + options", "nativeId": 138, "classification": "standard", "status": "pending", "blockedBy": [1]}, {"id": 3, "subject": "K3: KpiHistory project scaffold + options", "nativeId": 138, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 4, "subject": "K4: KpiHistoryRecorderActor", "nativeId": 139, "classification": "high-risk", "status": "pending", "blockedBy": [3]}, {"id": 4, "subject": "K4: KpiHistoryRecorderActor", "nativeId": 139, "classification": "high-risk", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "K5: Host wiring + appsettings", "nativeId": 140, "classification": "high-risk", "status": "pending", "blockedBy": [4]}, {"id": 5, "subject": "K5: Host wiring + appsettings", "nativeId": 140, "classification": "high-risk", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "K6: NotificationOutboxKpiSampleSource", "nativeId": 141, "classification": "small", "status": "pending", "blockedBy": [1]}, {"id": 6, "subject": "K6: NotificationOutboxKpiSampleSource", "nativeId": 141, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "K7: SiteCallAuditKpiSampleSource", "nativeId": 142, "classification": "small", "status": "pending", "blockedBy": [1]}, {"id": 7, "subject": "K7: SiteCallAuditKpiSampleSource", "nativeId": 142, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "K8: AuditLogKpiSampleSource", "nativeId": 143, "classification": "small", "status": "pending", "blockedBy": [1]}, {"id": 8, "subject": "K8: AuditLogKpiSampleSource", "nativeId": 143, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "K9: SiteHealthKpiSampleSource", "nativeId": 144, "classification": "standard", "status": "pending", "blockedBy": [1]}, {"id": 9, "subject": "K9: SiteHealthKpiSampleSource", "nativeId": 144, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "K10: KpiSeriesBucketer (pure helper)", "nativeId": 145, "classification": "small", "status": "pending", "blockedBy": [1]}, {"id": 10, "subject": "K10: KpiSeriesBucketer (pure helper)", "nativeId": 145, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 11, "subject": "K11: KpiHistoryQueryService (CentralUI)", "nativeId": 146, "classification": "standard", "status": "pending", "blockedBy": [2, 10]}, {"id": 11, "subject": "K11: KpiHistoryQueryService (CentralUI)", "nativeId": 146, "classification": "standard", "status": "completed", "blockedBy": [2, 10]},
{"id": 12, "subject": "K12: KpiTrendChart.razor reusable SVG component", "nativeId": 147, "classification": "standard", "status": "pending", "blockedBy": [1]}, {"id": 12, "subject": "K12: KpiTrendChart.razor reusable SVG component", "nativeId": 147, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 13, "subject": "K13: Notification Outbox page trend section", "nativeId": 148, "classification": "standard", "status": "pending", "blockedBy": [11, 12]}, {"id": 13, "subject": "K13: Notification Outbox page trend section", "nativeId": 148, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 14, "subject": "K14: Site Calls page trend section", "nativeId": 149, "classification": "standard", "status": "pending", "blockedBy": [11, 12]}, {"id": 14, "subject": "K14: Site Calls page trend section", "nativeId": 149, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 15, "subject": "K15: Audit Log page trend section", "nativeId": 150, "classification": "standard", "status": "pending", "blockedBy": [11, 12]}, {"id": 15, "subject": "K15: Audit Log page trend section", "nativeId": 150, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 16, "subject": "K16: Health dashboard per-site trend panel", "nativeId": 151, "classification": "standard", "status": "pending", "blockedBy": [11, 12]}, {"id": 16, "subject": "K16: Health dashboard per-site trend panel", "nativeId": 151, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 17, "subject": "K17: Integration — docs, deploy, Playwright, full verification", "nativeId": 152, "classification": "high-risk", "status": "pending", "blockedBy": [5, 6, 7, 8, 9, 13, 14, 15, 16]} {"id": 17, "subject": "K17: Integration — docs, deploy, Playwright, full verification", "nativeId": 152, "classification": "high-risk", "status": "completed", "blockedBy": [5, 6, 7, 8, 9, 13, 14, 15, 16]}
], ],
"lastUpdated": "2026-06-17" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md.tasks.json
+3 -2
View File
@@ -11,7 +11,8 @@
], ],
"reviewPolishCommit": "8dcc55f", "reviewPolishCommit": "8dcc55f",
"followUps": [ "followUps": [
{"id": 162, "subject": "normalize routed GetAttributes List values for cross-process transport (surfaced by WS-4 review)"} {"id": 162, "subject": "normalize routed GetAttributes List values for cross-process transport (surfaced by WS-4 review)", "status": "resolved", "note": "RESOLVED — DeploymentManagerActor.cs RouteInboundApiGetAttributes now normalizes via NormalizeRoutedReturnValue"}
], ],
"lastUpdated": "2026-06-17" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md
+3 -1
View File
@@ -239,7 +239,9 @@ fully cover the DELMIA/MES use case.
> **Status:** IMPLEMENTED. `Route.To(code).WaitForAttribute(name, targetValue, timeout)` is wired > **Status:** IMPLEMENTED. `Route.To(code).WaitForAttribute(name, targetValue, timeout)` is wired
> end-to-end (`RouteToWaitForAttributeRequest/Response` → `IInstanceRouter` → `CommunicationService` > end-to-end (`RouteToWaitForAttributeRequest/Response` → `IInstanceRouter` → `CommunicationService`
> → `SiteCommunicationActor` → `DeploymentManagerActor` → `InstanceActor`), value-equality only > → `SiteCommunicationActor` → `DeploymentManagerActor` → `InstanceActor`), value-equality only
> across the wire. NOT wired into the CentralUI Test-Run sandbox — that remains a follow-up. > across the wire. ~~NOT wired into the CentralUI Test-Run sandbox — that remains a follow-up.~~
> **Follow-up SHIPPED 2026-07-10** — the CentralUI Test-Run sandbox is wired via
> `SandboxInstanceGateway`/`SandboxScriptHost` (deferred-work register row #14 resolved).
--- ---
docs/plans/2026-06-17-waitfor-deferred-items.md.tasks.json
+6 -5
View File
@@ -1,10 +1,11 @@
{ {
"planPath": "docs/plans/2026-06-17-waitfor-deferred-items.md", "planPath": "docs/plans/2026-06-17-waitfor-deferred-items.md",
"tasks": [ "tasks": [
{"id": 1, "subject": "WD-1: site-local WaitForAsync + WaitResult + quality-gated mode (§3+§4.2)", "classification": "high-risk", "status": "pending", "parallelizableWith": [2]}, {"id": 1, "subject": "WD-1: site-local WaitForAsync + WaitResult + quality-gated mode (§3+§4.2)", "classification": "high-risk", "status": "completed", "parallelizableWith": [2]},
{"id": 2, "subject": "WD-2a: routed contract + central path (§6 part 1)", "classification": "high-risk", "status": "pending", "parallelizableWith": [1]}, {"id": 2, "subject": "WD-2a: routed contract + central path (§6 part 1)", "classification": "high-risk", "status": "completed", "parallelizableWith": [1]},
{"id": 3, "subject": "WD-2b: site unpacking + DeploymentManager handler (§6 part 2)", "classification": "high-risk", "status": "pending", "blockedBy": [2]}, {"id": 3, "subject": "WD-2b: site unpacking + DeploymentManager handler (§6 part 2)", "classification": "high-risk", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "WD-3: integration — docs + full verification", "classification": "standard", "status": "pending", "blockedBy": [1, 2, 3]} {"id": 4, "subject": "WD-3: integration — docs + full verification", "classification": "standard", "status": "completed", "blockedBy": [1, 2, 3]}
], ],
"lastUpdated": "2026-06-17" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-18-m10-uiux-platform.md
+8 -8
View File
@@ -459,15 +459,15 @@ Markup: two `<input type="datetime-local">` with `id="@(IdPrefix)-from"`/`-to`,
## Deferred / out of scope (log as FOLLOWUPs at INT) ## Deferred / out of scope (log as FOLLOWUPs at INT)
- Unified offset+keyset pagination framework (blocked by total-count mismatch). - Unified offset+keyset pagination framework (blocked by total-count mismatch).
- `Deployments.razor` PagerWindow → OffsetPager (different windowed UX; intentionally left). - `Deployments.razor` PagerWindow → OffsetPager (different windowed UX; intentionally left).
- Complex `TemplateEdit` page-embedded modals → host migration. - ~~Complex `TemplateEdit` page-embedded modals → host migration.~~ **DONE 2026-08-01.**
- TreeView arrow-key navigation (R7). - ~~TreeView arrow-key navigation (R7).~~ **DONE 2026-08-01.**
- Theme-package side-rail dark theming, IF the spike verdict is "rail stays light" (coordination follow-up). - Theme-package side-rail dark theming, IF the spike verdict is "rail stays light" (coordination follow-up).
## Follow-ups logged at delivery (INT findings) ## Follow-ups logged at delivery (INT findings)
- **#207 (pre-existing, open since M6/K14):** `QueryStringDrillInTests` fixture does not register `IKpiHistoryQueryService` — 3 `SiteCallsReport` drill-in tests red on this gap; unrelated to M10. - **#207 — FIXED (2026-08-01 bookkeeping sync).** ~~(pre-existing, open since M6/K14):~~ `QueryStringDrillInTests` now registers the `IKpiHistoryQueryService` substitute in its fixture (lines ~165-167), so the 3 `SiteCallsReport` drill-in tests are green.
- **#163 (pre-existing):** `InstanceConfigureListOverrideTests` codec roundtrip red; pre-dates M10. - **#163 — CLOSED (2026-08-01).** ~~(pre-existing):~~ `InstanceConfigureListOverrideTests` codec roundtrip verified green (6/6; `dotnet test …CentralUI.Tests --filter InstanceConfigureListOverride`) — fixed at some point since it was logged; no longer a known red. (#207 `QueryStringDrillInTests` re-verified green the same day, 4/4.)
- **NotificationReport `OffsetPager` always-visible:** pager is now always visible when results exist (previously hidden on sub-page-size sets); buttons are correctly disabled on a single page — product decision whether to re-add an `@if (_totalCount > _pageSize)` guard. - **NotificationReport `OffsetPager` always-visible — DECIDED + DONE (2026-08-01).** A pager whose only controls are permanently disabled is noise, so the conventional behaviour ships: the bar is hidden on a provably single page. Implemented **reusably but opt-in** — `OffsetPager.HideWhenSinglePage` (default `false`, so no consumer changes behaviour implicitly), set `true` at the `NotificationReport` call site. It is NOT defaulted on because the pager's summary span doubles as the "N total" result-count readout on `ConfigurationAuditLog`, which has no other place to show it (and whose Playwright fixture asserts `3 total` on a 3-row single page). The guard only fires when single-page-ness is *positively* established (`PageCount <= 1` **and** `Page <= 1` **and** `!HasNextPage`), so a null `TotalCount` still renders the bar.
- **`Deployments.razor` PagerWindow intentionally kept:** windowed numbered-button UX is deliberate; NOT migrated to `OffsetPager`. - **`Deployments.razor` PagerWindow intentionally kept:** windowed numbered-button UX is deliberate; NOT migrated to `OffsetPager`.
- **`TemplateEdit` inline modals NOT migrated:** the page-embedded modals (wave-3 T34c already tokenized their backdrops); full migration to the host is deferred. - **`TemplateEdit` inline modals — MIGRATED (2026-08-01).** All four page-embedded modals (Attribute, Alarm, Native Alarm Source, Script) now open through `IDialogService.ShowAsync`, so the host owns the backdrop, focus trap, Escape and focus restoration. Each body was extracted to its own component next to the page — `TemplateAttributeDialog`, `TemplateAlarmDialog`, `TemplateNativeAlarmSourceDialog`, `TemplateScriptDialog` — following the `MoveDataConnectionDialog` pattern: the body owns the form state and renders validation/server errors INLINE while staying open, closing with `Close(true)` only on a successful save. Persistence stayed on the page (it owns `TemplateService` and the inherited-member rules) and is reached through an `OnSaveAsync` delegate returning `null` on success or the message to display. The script dialog keeps all four tab panels mounted (Monaco/JSONJoy must not tear down on tab switch) and hosts the Test Run panel, which needs the live unsaved buffer. Extraction moved markup that three structural source-scanning tests pinned (`TemplateNativeAlarmSourceEditorTests`, `AttributeListEditorTests`, `TestRunWarningTests`) — all three were repointed at the new files.
- **TreeView full arrow-key navigation (R7) still deferred.** - **~~TreeView full arrow-key navigation (R7) still deferred.~~ DONE 2026-08-01** — the WAI-ARIA tree pattern (roving tabindex, Arrow/Home/End movement, Enter/Space activation, `aria-level`) now ships on `TreeView<TItem>`; see `docs/components/TreeView.md`.
- **Full-app `bg-light`/`bg-white` → theme-aware utility sweep deferred:** only the bounded modal-surface offenders were addressed in T34c; INT dark-mode smoke may surface additional instances. - **Full-app `bg-light`/`bg-white` → theme-aware utility sweep — DONE 2026-08-01.** 35 class swaps across 19 Central UI files: surfaces/`<pre>`/`<code>` `bg-light` → `bg-body-secondary`, panel `bg-white` → `bg-body`, neutral `badge bg-light text-dark` → `badge bg-secondary-subtle text-secondary-emphasis`, `bg-light text-muted border` → `bg-body-secondary text-body-secondary border`. **Deliberately left** (7 sites): the neutral member of a *status-badge* switch/ternary whose siblings are all solid non-theme-aware colours (`bg-success`/`bg-danger`/`bg-warning`), where swapping only the neutral one breaks the set's visual weight — `Topology.razor:513,588`, `InstanceConfigure.razor:1520`, `NotificationReport.razor` `StatusBadgeClass` fallback, `TransportImport.razor` ConflictKind fallback, `SecuredWrites.razor` `text-bg-light` fallback, `Health.razor:377`. `SchemaBuilder.razor:88` was already `bg-light-subtle` (theme-aware). `bg-dark text-light` console panels and `site.css`/`#reconnect-modal` are intentional fixed colours.
docs/plans/2026-06-18-m10-uiux-platform.md.tasks.json
+19 -18
View File
@@ -1,23 +1,24 @@
{ {
"planPath": "docs/plans/2026-06-18-m10-uiux-platform.md", "planPath": "docs/plans/2026-06-18-m10-uiux-platform.md",
"tasks": [ "tasks": [
{"id": 269, "subject": "M10-T33a: Modal host ShowAsync<T> + focus trap/restore + backdrop hook", "status": "pending"}, {"id": 269, "subject": "M10-T33a: Modal host ShowAsync<T> + focus trap/restore + backdrop hook", "status": "completed"},
{"id": 270, "subject": "M10-T34-spike: Verify ZB.MOM.WW.Theme dark-mode feasibility", "status": "pending"}, {"id": 270, "subject": "M10-T34-spike: Verify ZB.MOM.WW.Theme dark-mode feasibility", "status": "completed"},
{"id": 271, "subject": "M10-T41: Alarm-override Playwright trigger-config scenarios", "status": "pending"}, {"id": 271, "subject": "M10-T41: Alarm-override Playwright trigger-config scenarios", "status": "completed"},
{"id": 272, "subject": "M10-T33b: Migrate 5 simple dialogs to ShowAsync<T>", "status": "pending", "blockedBy": [269]}, {"id": 272, "subject": "M10-T33b: Migrate 5 simple dialogs to ShowAsync<T>", "status": "completed", "blockedBy": [269]},
{"id": 273, "subject": "M10-T34a: Dark token layer in site.css + .sb-modal-backdrop", "status": "pending", "blockedBy": [270]}, {"id": 273, "subject": "M10-T34a: Dark token layer in site.css + .sb-modal-backdrop", "status": "completed", "blockedBy": [270]},
{"id": 274, "subject": "M10-T35a: OffsetPager component", "status": "pending"}, {"id": 274, "subject": "M10-T35a: OffsetPager component", "status": "completed"},
{"id": 275, "subject": "M10-T35b: KeysetPager component", "status": "pending"}, {"id": 275, "subject": "M10-T35b: KeysetPager component", "status": "completed"},
{"id": 276, "subject": "M10-T35c: DateTimeRangeFilter component", "status": "pending"}, {"id": 276, "subject": "M10-T35c: DateTimeRangeFilter component", "status": "completed"},
{"id": 277, "subject": "M10-T35d: Adopt OffsetPager+DateTimeRangeFilter into NotificationReport (+backdrop token)", "status": "pending", "blockedBy": [274, 276, 273]}, {"id": 277, "subject": "M10-T35d: Adopt OffsetPager+DateTimeRangeFilter into NotificationReport (+backdrop token)", "status": "completed", "blockedBy": [274, 276, 273]},
{"id": 278, "subject": "M10-T35e: Adopt OffsetPager+DateTimeRangeFilter into ConfigurationAuditLog", "status": "pending", "blockedBy": [274, 276]}, {"id": 278, "subject": "M10-T35e: Adopt OffsetPager+DateTimeRangeFilter into ConfigurationAuditLog", "status": "completed", "blockedBy": [274, 276]},
{"id": 279, "subject": "M10-T35f: Adopt KeysetPager+DateTimeRangeFilter into SiteCallsReport (+backdrop token)", "status": "pending", "blockedBy": [275, 276, 273]}, {"id": 279, "subject": "M10-T35f: Adopt KeysetPager+DateTimeRangeFilter into SiteCallsReport (+backdrop token)", "status": "completed", "blockedBy": [275, 276, 273]},
{"id": 280, "subject": "M10-T35g: Adopt KeysetPager into AuditResultsGrid + DateTimeRangeFilter into AuditFilterBar", "status": "pending", "blockedBy": [275, 276]}, {"id": 280, "subject": "M10-T35g: Adopt KeysetPager into AuditResultsGrid + DateTimeRangeFilter into AuditFilterBar", "status": "completed", "blockedBy": [275, 276]},
{"id": 281, "subject": "M10-T35h: Adopt DateTimeRangeFilter into EventLogs", "status": "pending", "blockedBy": [276]}, {"id": 281, "subject": "M10-T35h: Adopt DateTimeRangeFilter into EventLogs", "status": "completed", "blockedBy": [276]},
{"id": 282, "subject": "M10-T34c: Tokenize remaining standalone backdrops + bg-* audit", "status": "pending", "blockedBy": [273]}, {"id": 282, "subject": "M10-T34c: Tokenize remaining standalone backdrops + bg-* audit", "status": "completed", "blockedBy": [273]},
{"id": 283, "subject": "M10-T34b: Dark-mode toggle + persistence + SSR hydration", "status": "pending", "blockedBy": [270, 273]}, {"id": 283, "subject": "M10-T34b: Dark-mode toggle + persistence + SSR hydration", "status": "completed", "blockedBy": [270, 273]},
{"id": 284, "subject": "M10-T36a: Accessibility pass — TreeView chevron + icon audit + toast verify", "status": "pending"}, {"id": 284, "subject": "M10-T36a: Accessibility pass — TreeView chevron + icon audit + toast verify", "status": "completed"},
{"id": 285, "subject": "M10-INT: Integration — build, docker, Playwright, a11y+dark smoke, docs, review", "status": "pending", "blockedBy": [271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284]} {"id": 285, "subject": "M10-INT: Integration — build, docker, Playwright, a11y+dark smoke, docs, review", "status": "completed", "blockedBy": [271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284]}
], ],
"lastUpdated": "2026-06-18" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-18-m7-opcua-mxgateway-ux.md.tasks.json
+24 -23
View File
@@ -4,28 +4,28 @@
"branch": "worktree-m7-opcua-mxgateway-ux", "branch": "worktree-m7-opcua-mxgateway-ux",
"baseSha": "254e0e7", "baseSha": "254e0e7",
"tasks": [ "tasks": [
{"id": 184, "subject": "M7-A1: Extract AlarmStateBadges shared component", "classification": "standard", "status": "pending"}, {"id": 184, "subject": "M7-A1: Extract AlarmStateBadges shared component", "classification": "standard", "status": "completed"},
{"id": 185, "subject": "M7-A2: Operator Alarm Summary page + fan-out service", "classification": "standard", "status": "pending", "blockedBy": [184]}, {"id": 185, "subject": "M7-A2: Operator Alarm Summary page + fan-out service", "classification": "standard", "status": "completed", "blockedBy": [184]},
{"id": 186, "subject": "M7-A3: Operator + Verifier roles + policies + LDAP mapping", "classification": "high-risk", "status": "pending"}, {"id": 186, "subject": "M7-A3: Operator + Verifier roles + policies + LDAP mapping", "classification": "high-risk", "status": "completed"},
{"id": 187, "subject": "M7-B1: Browse type-info fields on BrowseNode", "classification": "standard", "status": "pending"}, {"id": 187, "subject": "M7-B1: Browse type-info fields on BrowseNode", "classification": "standard", "status": "completed"},
{"id": 188, "subject": "M7-B2: BrowseNext continuation through browse contract", "classification": "high-risk", "status": "pending", "blockedBy": [187]}, {"id": 188, "subject": "M7-B2: BrowseNext continuation through browse contract", "classification": "high-risk", "status": "completed", "blockedBy": [187]},
{"id": 189, "subject": "M7-B3: Thread continuation token through browse plumbing", "classification": "standard", "status": "pending", "blockedBy": [188]}, {"id": 189, "subject": "M7-B3: Thread continuation token through browse plumbing", "classification": "standard", "status": "completed", "blockedBy": [188]},
{"id": 190, "subject": "M7-B4: Bounded recursive address-space search — adapter", "classification": "high-risk", "status": "pending", "blockedBy": [188]}, {"id": 190, "subject": "M7-B4: Bounded recursive address-space search — adapter", "classification": "high-risk", "status": "completed", "blockedBy": [188]},
{"id": 191, "subject": "M7-B5: Search plumbing — message + actor + comm + service", "classification": "standard", "status": "pending", "blockedBy": [189, 190]}, {"id": 191, "subject": "M7-B5: Search plumbing — message + actor + comm + service", "classification": "standard", "status": "completed", "blockedBy": [189, 190]},
{"id": 192, "subject": "M7-B6: NodeBrowserDialog — load-more + search box + type column", "classification": "standard", "status": "pending", "blockedBy": [189, 191]}, {"id": 192, "subject": "M7-B6: NodeBrowserDialog — load-more + search box + type column", "classification": "standard", "status": "completed", "blockedBy": [189, 191]},
{"id": 193, "subject": "M7-B7: Verify-endpoint — message + site probe handler", "classification": "high-risk", "status": "pending", "blockedBy": [191]}, {"id": 193, "subject": "M7-B7: Verify-endpoint — message + site probe handler", "classification": "high-risk", "status": "completed", "blockedBy": [191]},
{"id": 194, "subject": "M7-B8: Verify-endpoint plumbing + UI", "classification": "standard", "status": "pending", "blockedBy": [193]}, {"id": 194, "subject": "M7-B8: Verify-endpoint plumbing + UI", "classification": "standard", "status": "completed", "blockedBy": [193]},
{"id": 195, "subject": "M7-B9: Cert trust — per-node CertStore actor + broadcast", "classification": "high-risk", "status": "pending", "blockedBy": [193]}, {"id": 195, "subject": "M7-B9: Cert trust — per-node CertStore actor + broadcast", "classification": "high-risk", "status": "completed", "blockedBy": [193]},
{"id": 196, "subject": "M7-B10: Cert trust plumbing + cert-management UI", "classification": "standard", "status": "pending", "blockedBy": [194, 195]}, {"id": 196, "subject": "M7-B10: Cert trust plumbing + cert-management UI", "classification": "standard", "status": "completed", "blockedBy": [194, 195]},
{"id": 197, "subject": "M7-C1: PendingSecuredWrite entity + persistence + migration", "classification": "high-risk", "status": "pending"}, {"id": 197, "subject": "M7-C1: PendingSecuredWrite entity + persistence + migration", "classification": "high-risk", "status": "completed"},
{"id": 198, "subject": "M7-C2: Secured-write commands + submit/reject/list handlers", "classification": "high-risk", "status": "pending", "blockedBy": [197, 186]}, {"id": 198, "subject": "M7-C2: Secured-write commands + submit/reject/list handlers", "classification": "high-risk", "status": "completed", "blockedBy": [197, 186]},
{"id": 199, "subject": "M7-C3: Approve → site write relay", "classification": "high-risk", "status": "pending", "blockedBy": [198]}, {"id": 199, "subject": "M7-C3: Approve → site write relay", "classification": "high-risk", "status": "completed", "blockedBy": [198]},
{"id": 200, "subject": "M7-C4: AuditKind.SecuredWrite + audit wiring", "classification": "high-risk", "status": "pending", "blockedBy": [198, 199]}, {"id": 200, "subject": "M7-C4: AuditKind.SecuredWrite + audit wiring", "classification": "high-risk", "status": "completed", "blockedBy": [198, 199]},
{"id": 201, "subject": "M7-C5: Secured Writes Central UI page", "classification": "standard", "status": "pending", "blockedBy": [198, 199, 186]}, {"id": 201, "subject": "M7-C5: Secured Writes Central UI page", "classification": "standard", "status": "completed", "blockedBy": [198, 199, 186]},
{"id": 202, "subject": "M7-D1: OverrideCsvParser pure helper", "classification": "standard", "status": "pending"}, {"id": 202, "subject": "M7-D1: OverrideCsvParser pure helper", "classification": "standard", "status": "completed"},
{"id": 203, "subject": "M7-D2: InstanceConfigure CSV import UI", "classification": "standard", "status": "pending", "blockedBy": [202]}, {"id": 203, "subject": "M7-D2: InstanceConfigure CSV import UI", "classification": "standard", "status": "completed", "blockedBy": [202]},
{"id": 204, "subject": "M7-D3: CLI instance import-overrides --file", "classification": "small", "status": "pending", "blockedBy": [202]}, {"id": 204, "subject": "M7-D3: CLI instance import-overrides --file", "classification": "small", "status": "completed", "blockedBy": [202]},
{"id": 205, "subject": "M7-E1: Integration — docs, full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "pending", "blockedBy": [184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204]} {"id": 205, "subject": "M7-E1: Integration — docs, full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "completed", "blockedBy": [184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204]}
], ],
"waves": { "waves": {
"A": [184, 185, 186], "A": [184, 185, 186],
@@ -34,5 +34,6 @@
"D": [202, 203, 204], "D": [202, 203, 204],
"E": [205] "E": [205]
}, },
"lastUpdated": "2026-06-18" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-18-m8-transport.md.tasks.json
+17 -16
View File
@@ -1,21 +1,22 @@
{ {
"planPath": "docs/plans/2026-06-18-m8-transport.md", "planPath": "docs/plans/2026-06-18-m8-transport.md",
"tasks": [ "tasks": [
{"id": 210, "subject": "M8-A1: Commons name-map types + preview/selection/summary extensions", "status": "pending"}, {"id": 210, "subject": "M8-A1: Commons name-map types + preview/selection/summary extensions", "status": "completed"},
{"id": 211, "subject": "M8-A2: Transport bundle DTOs — Site/DataConnection/Instance + BundleContentDto", "status": "pending"}, {"id": 211, "subject": "M8-A2: Transport bundle DTOs — Site/DataConnection/Instance + BundleContentDto", "status": "completed"},
{"id": 212, "subject": "M8-A3: LineDiffer — pure Myers per-line diff helper (T20)", "status": "pending"}, {"id": 212, "subject": "M8-A3: LineDiffer — pure Myers per-line diff helper (T20)", "status": "completed"},
{"id": 213, "subject": "M8-B1: DependencyResolver site/instance expansion", "status": "pending", "blockedBy": [211]}, {"id": 213, "subject": "M8-B1: DependencyResolver site/instance expansion", "status": "completed", "blockedBy": [211]},
{"id": 214, "subject": "M8-B2: EntitySerializer site/connection/instance mapping (both directions)", "status": "pending", "blockedBy": [211]}, {"id": 214, "subject": "M8-B2: EntitySerializer site/connection/instance mapping (both directions)", "status": "completed", "blockedBy": [211]},
{"id": 215, "subject": "M8-B3: ManifestBuilder + summary counts + schemaVersion 1.1", "status": "pending", "blockedBy": [210]}, {"id": 215, "subject": "M8-B3: ManifestBuilder + summary counts + schemaVersion 1.1", "status": "completed", "blockedBy": [210]},
{"id": 216, "subject": "M8-B4: Export plumbing — selection wiring, command, ManagementActor, CLI", "status": "pending", "blockedBy": [213, 214, 215]}, {"id": 216, "subject": "M8-B4: Export plumbing — selection wiring, command, ManagementActor, CLI", "status": "completed", "blockedBy": [213, 214, 215]},
{"id": 217, "subject": "M8-C1: ArtifactDiff — Myers code diff (T20) + Site/Connection/Instance compares", "status": "pending", "blockedBy": [211, 212]}, {"id": 217, "subject": "M8-C1: ArtifactDiff — Myers code diff (T20) + Site/Connection/Instance compares", "status": "completed", "blockedBy": [211, 212]},
{"id": 218, "subject": "M8-C2: BundleImporter.PreviewAsync — new-type diff + required-mapping detection + blockers", "status": "pending", "blockedBy": [214, 217]}, {"id": 218, "subject": "M8-C2: BundleImporter.PreviewAsync — new-type diff + required-mapping detection + blockers", "status": "completed", "blockedBy": [214, 217]},
{"id": 219, "subject": "M8-D1: BundleImporter.ApplyAsync — nameMap, resolve-or-create, instance upsert + FK rewire", "status": "pending", "blockedBy": [218]}, {"id": 219, "subject": "M8-D1: BundleImporter.ApplyAsync — nameMap, resolve-or-create, instance upsert + FK rewire", "status": "completed", "blockedBy": [218]},
{"id": 220, "subject": "M8-D2: #16 — real stale-instance enumeration in ImportResult", "status": "pending", "blockedBy": [219]}, {"id": 220, "subject": "M8-D2: #16 — real stale-instance enumeration in ImportResult", "status": "completed", "blockedBy": [219]},
{"id": 221, "subject": "M8-D3: Import plumbing — name-map through command, ManagementActor, CLI", "status": "pending", "blockedBy": [219, 216]}, {"id": 221, "subject": "M8-D3: Import plumbing — name-map through command, ManagementActor, CLI", "status": "completed", "blockedBy": [219, 216]},
{"id": 222, "subject": "M8-E1: Export wizard — Sites/Instances selection", "status": "pending", "blockedBy": [216]}, {"id": 222, "subject": "M8-E1: Export wizard — Sites/Instances selection", "status": "completed", "blockedBy": [216]},
{"id": 223, "subject": "M8-E2: Import wizard — Map step + Modified +/- diff render", "status": "pending", "blockedBy": [221]}, {"id": 223, "subject": "M8-E2: Import wizard — Map step + Modified +/- diff render", "status": "completed", "blockedBy": [221]},
{"id": 224, "subject": "M8-INT: Docs, full build, docker rebuild, Playwright, live smoke, end-to-end trace", "status": "pending", "blockedBy": [222, 223]} {"id": 224, "subject": "M8-INT: Docs, full build, docker rebuild, Playwright, live smoke, end-to-end trace", "status": "completed", "blockedBy": [222, 223]}
], ],
"lastUpdated": "2026-06-18" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-18-m9-templates-authoring.md.tasks.json
+19 -18
View File
@@ -4,23 +4,23 @@
"branch": "worktree-m9-templates-authoring", "branch": "worktree-m9-templates-authoring",
"baseRef": "72aec3b4", "baseRef": "72aec3b4",
"tasks": [ "tasks": [
{"id": 240, "label": "T22", "subject": "M9-T22: Template tree search box", "classification": "small", "wave": 1, "status": "pending"}, {"id": 240, "label": "T22", "subject": "M9-T22: Template tree search box", "classification": "small", "wave": 1, "status": "completed"},
{"id": 241, "label": "CLI", "subject": "M9-CLI: cached-call retry/discard CLI group", "classification": "small", "wave": 1, "status": "pending"}, {"id": 241, "label": "CLI", "subject": "M9-CLI: cached-call retry/discard CLI group", "classification": "small", "wave": 1, "status": "completed"},
{"id": 242, "label": "T28a", "subject": "M9-T28a: Strict expression-trigger kind — backend", "classification": "small", "wave": 1, "status": "pending"}, {"id": 242, "label": "T28a", "subject": "M9-T28a: Strict expression-trigger kind — backend", "classification": "small", "wave": 1, "status": "completed"},
{"id": 243, "label": "T28b", "subject": "M9-T28b: Strict trigger-kind — UI selector + CLI flag", "classification": "small", "wave": 1, "status": "pending", "blockedBy": [242]}, {"id": 243, "label": "T28b", "subject": "M9-T28b: Strict trigger-kind — UI selector + CLI flag", "classification": "small", "wave": 1, "status": "completed", "blockedBy": [242]},
{"id": 244, "label": "T23a", "subject": "M9-T23a: Folder sibling reorder — service + command + handler", "classification": "standard", "wave": 2, "status": "pending"}, {"id": 244, "label": "T23a", "subject": "M9-T23a: Folder sibling reorder — service + command + handler", "classification": "standard", "wave": 2, "status": "completed"},
{"id": 245, "label": "T23b", "subject": "M9-T23b: Folder reorder + root context menu — UI", "classification": "standard", "wave": 2, "status": "pending", "blockedBy": [240, 244]}, {"id": 245, "label": "T23b", "subject": "M9-T23b: Folder reorder + root context menu — UI", "classification": "standard", "wave": 2, "status": "completed", "blockedBy": [240, 244]},
{"id": 246, "label": "T25", "subject": "M9-T25: Connection live-status indicators", "classification": "standard", "wave": 2, "status": "pending"}, {"id": 246, "label": "T25", "subject": "M9-T25: Connection live-status indicators", "classification": "standard", "wave": 2, "status": "completed"},
{"id": 247, "label": "T24a", "subject": "M9-T24a: Move data connection between sites — command + handler + guards", "classification": "high-risk", "wave": 3, "status": "pending"}, {"id": 247, "label": "T24a", "subject": "M9-T24a: Move data connection between sites — command + handler + guards", "classification": "high-risk", "wave": 3, "status": "completed"},
{"id": 248, "label": "T24b", "subject": "M9-T24b: Move connection — UI dialog + action", "classification": "standard", "wave": 3, "status": "pending", "blockedBy": [247, 246]}, {"id": 248, "label": "T24b", "subject": "M9-T24b: Move connection — UI dialog + action", "classification": "standard", "wave": 3, "status": "completed", "blockedBy": [247, 246]},
{"id": 249, "label": "T32a", "subject": "M9-T32a: SharedSchema entity + EF config + migration + repo", "classification": "high-risk", "wave": 3, "status": "pending"}, {"id": 249, "label": "T32a", "subject": "M9-T32a: SharedSchema entity + EF config + migration + repo", "classification": "high-risk", "wave": 3, "status": "completed"},
{"id": 250, "label": "T32b", "subject": "M9-T32b: JSON Schema $ref resolver + deploy-time validation", "classification": "high-risk", "wave": 3, "status": "pending", "blockedBy": [249]}, {"id": 250, "label": "T32b", "subject": "M9-T32b: JSON Schema $ref resolver + deploy-time validation", "classification": "high-risk", "wave": 3, "status": "completed", "blockedBy": [249]},
{"id": 251, "label": "T32c", "subject": "M9-T32c: Schema library — CRUD commands + handlers + Central UI page", "classification": "high-risk", "wave": 4, "status": "pending", "blockedBy": [249, 250]}, {"id": 251, "label": "T32c", "subject": "M9-T32c: Schema library — CRUD commands + handlers + Central UI page", "classification": "high-risk", "wave": 4, "status": "completed", "blockedBy": [249, 250]},
{"id": 252, "label": "T30", "subject": "M9-T30: Schema-driven nested value-entry forms", "classification": "standard", "wave": 4, "status": "pending", "blockedBy": [250]}, {"id": 252, "label": "T30", "subject": "M9-T30: Schema-driven nested value-entry forms", "classification": "standard", "wave": 4, "status": "completed", "blockedBy": [250]},
{"id": 253, "label": "T31", "subject": "M9-T31: Monaco JSON-Schema hover/completion", "classification": "standard", "wave": 4, "status": "pending", "blockedBy": [250, 252]}, {"id": 253, "label": "T31", "subject": "M9-T31: Monaco JSON-Schema hover/completion", "classification": "standard", "wave": 4, "status": "completed", "blockedBy": [250, 252]},
{"id": 254, "label": "T26a", "subject": "M9-T26a: Inheritance resolve service + query command", "classification": "high-risk", "wave": 4, "status": "pending"}, {"id": 254, "label": "T26a", "subject": "M9-T26a: Inheritance resolve service + query command", "classification": "high-risk", "wave": 4, "status": "completed"},
{"id": 255, "label": "T26b", "subject": "M9-T26b: TemplateEdit — full inherited set + staleness banner", "classification": "standard", "wave": 4, "status": "pending", "blockedBy": [254]}, {"id": 255, "label": "T26b", "subject": "M9-T26b: TemplateEdit — full inherited set + staleness banner", "classification": "standard", "wave": 4, "status": "completed", "blockedBy": [254]},
{"id": 256, "label": "INT", "subject": "M9-INT: Integration — build, docker, Playwright, smoke, end-to-end trace", "classification": "high-risk", "wave": 5, "status": "pending", "blockedBy": [240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]} {"id": 256, "label": "INT", "subject": "M9-INT: Integration — build, docker, Playwright, smoke, end-to-end trace", "classification": "high-risk", "wave": 5, "status": "completed", "blockedBy": [240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}
], ],
"serializationPoints": { "serializationPoints": {
"ManagementActor.cs+ManagementCommandRegistry.cs": [244, 247, 251, 254], "ManagementActor.cs+ManagementCommandRegistry.cs": [244, 247, 251, 254],
@@ -30,5 +30,6 @@
"TemplateEdit.razor": [243, 255], "TemplateEdit.razor": [243, 255],
"ParameterValueForm.razor/value-entry-surface": [252, 253] "ParameterValueForm.razor/value-entry-surface": [252, 253]
}, },
"lastUpdated": "2026-06-18" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-19-sms-notifications.md.tasks.json
+13 -12
View File
@@ -1,17 +1,18 @@
{ {
"planPath": "docs/plans/2026-06-19-sms-notifications.md", "planPath": "docs/plans/2026-06-19-sms-notifications.md",
"tasks": [ "tasks": [
{"id": 299, "subject": "S1: Commons foundation — NotificationType.Sms + recipient phone + SmsConfiguration + repo iface", "status": "pending"}, {"id": 299, "subject": "S1: Commons foundation — NotificationType.Sms + recipient phone + SmsConfiguration + repo iface", "status": "completed"},
{"id": 300, "subject": "S2: Config-DB — EF mappings + AuthToken encryption + idempotent migration", "status": "pending", "blockedBy": [299]}, {"id": 300, "subject": "S2: Config-DB — EF mappings + AuthToken encryption + idempotent migration", "status": "completed", "blockedBy": [299]},
{"id": 303, "subject": "S5: Management — list-command Type + SMS-config commands/handlers", "status": "pending", "blockedBy": [299]}, {"id": 303, "subject": "S5: Management — list-command Type + SMS-config commands/handlers", "status": "completed", "blockedBy": [299]},
{"id": 301, "subject": "S3: SmsNotificationDeliveryAdapter (Twilio REST) + classifier + options + DI + tests", "status": "pending", "blockedBy": [299, 300]}, {"id": 301, "subject": "S3: SmsNotificationDeliveryAdapter (Twilio REST) + classifier + options + DI + tests", "status": "completed", "blockedBy": [299, 300]},
{"id": 302, "subject": "S4: NotificationOutboxActor ingest type-stamping from list", "status": "pending", "blockedBy": [299]}, {"id": 302, "subject": "S4: NotificationOutboxActor ingest type-stamping from list", "status": "completed", "blockedBy": [299]},
{"id": 304, "subject": "S6: CLI — list --type/--phones + notification sms group", "status": "pending", "blockedBy": [303]}, {"id": 304, "subject": "S6: CLI — list --type/--phones + notification sms group", "status": "completed", "blockedBy": [303]},
{"id": 305, "subject": "S7: Central UI — NotificationListForm adapter-gated Type selector + per-type recipient input", "status": "pending", "blockedBy": [299, 303]}, {"id": 305, "subject": "S7: Central UI — NotificationListForm adapter-gated Type selector + per-type recipient input", "status": "completed", "blockedBy": [299, 303]},
{"id": 306, "subject": "S8: Central UI — NotificationLists Type column", "status": "pending", "blockedBy": [299]}, {"id": 306, "subject": "S8: Central UI — NotificationLists Type column", "status": "completed", "blockedBy": [299]},
{"id": 307, "subject": "S9: Central UI — SMS configuration page (/notifications/sms)", "status": "pending", "blockedBy": [300, 303]}, {"id": 307, "subject": "S9: Central UI — SMS configuration page (/notifications/sms)", "status": "completed", "blockedBy": [300, 303]},
{"id": 308, "subject": "S10: Transport — recipient PhoneNumber DTO + SmsConfigDto round-trip", "status": "pending", "blockedBy": [299]}, {"id": 308, "subject": "S10: Transport — recipient PhoneNumber DTO + SmsConfigDto round-trip", "status": "completed", "blockedBy": [299]},
{"id": 309, "subject": "S11: INT — build, drift, docker, Playwright, live smoke, docs, whole-branch review", "status": "pending", "blockedBy": [300, 301, 302, 303, 304, 305, 306, 307, 308]} {"id": 309, "subject": "S11: INT — build, drift, docker, Playwright, live smoke, docs, whole-branch review", "status": "completed", "blockedBy": [300, 301, 302, 303, 304, 305, 306, 307, 308]}
], ],
"lastUpdated": "2026-06-19" "lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
} }
docs/plans/2026-06-26-deploy-config-notify-and-fetch-design.md
+3 -1
View File
@@ -3,6 +3,8 @@
**Date:** 2026-06-26 · **Status:** APPROVED (design) · **Area:** Deployment Manager / Site Runtime / Cluster Communication **Date:** 2026-06-26 · **Status:** APPROVED (design) · **Area:** Deployment Manager / Site Runtime / Cluster Communication
**Fixes:** [`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`](../known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md) **Fixes:** [`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`](../known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md)
> **2026-08-01 bookkeeping sync:** the central→site ClusterClient hop described below is now **gRPC** (`SiteCommandService`/`GrpcSiteTransport`) after the 2026-07-22 ClusterClient→gRPC migration — the notify-and-fetch behavior described here is unchanged.
## 1. Problem ## 1. Problem
The flattened instance config travels over Akka on **two** hops today, and both are bounded by Akka.Remote's default `maximum-frame-size` (**128 KB / `128000b`**). A large config (e.g. a 3rd composition of the same base template) silently breaks both: The flattened instance config travels over Akka on **two** hops today, and both are bounded by Akka.Remote's default `maximum-frame-size` (**128 KB / `128000b`**). A large config (e.g. a 3rd composition of the same base template) silently breaks both:
@@ -177,7 +179,7 @@ Smoke-tested on the docker cluster (rebuilt from this branch). Validated end-to-
The smoke surfaced two real bugs in the reconciliation path (missed by unit/integration tests because those didn't have a second concurrent node or a lingering expired row), both fixed: The smoke surfaced two real bugs in the reconciliation path (missed by unit/integration tests because those didn't have a second concurrent node or a lingering expired row), both fixed:
1. **Concurrent-gap omit** — when two nodes were concurrently missing the same instance, the second node's `StagePendingIfAbsentAsync` returned false and the handler *omitted* the item, leaving that node unhealed. Fix: on false, return the **existing** pending row's deploymentId + token (multi-use within TTL) so all concurrently-missing nodes heal in the same round. 1. **Concurrent-gap omit** — when two nodes were concurrently missing the same instance, the second node's `StagePendingIfAbsentAsync` returned false and the handler *omitted* the item, leaving that node unhealed. Fix: on false, return the **existing** pending row's deploymentId + token (multi-use within TTL) so all concurrently-missing nodes heal in the same round.
2. **Expired pending row blocks self-heal** — `StagePendingIfAbsentAsync` checked existence by `InstanceId` ignoring expiry, so an expired-but-unpurged row (the periodic purge is still a deferred TODO) blocked a fresh stage *and* would collide with the snapshot's reused `DeploymentId` on the unique index. Fix: **expiry-aware staging** — delete expired rows for the instance first, then check only live rows; `GetPendingDeploymentByInstanceIdAsync` filters by expiry. This also opportunistically cleans expired rows, reducing reliance on the deferred periodic purge. 2. **Expired pending row blocks self-heal** — `StagePendingIfAbsentAsync` checked existence by `InstanceId` ignoring expiry, so an expired-but-unpurged row (the periodic purge was a deferred TODO at the time — **RESOLVED 2026-08-01 bookkeeping sync: `PendingDeploymentPurgeActor` shipped in PLAN-04; deferred-work register row #20 closed**) blocked a fresh stage *and* would collide with the snapshot's reused `DeploymentId` on the unique index. Fix: **expiry-aware staging** — delete expired rows for the instance first, then check only live rows; `GetPendingDeploymentByInstanceIdAsync` filters by expiry. This also opportunistically cleans expired rows, reducing reliance on the deferred periodic purge.
## 12. Affected files (for the plan) ## 12. Affected files (for the plan)
docs/plans/2026-06-30-mes-alarm-status-api.md
+79 -58
View File
@@ -1,8 +1,8 @@
# MES Alarm-Status API — Implementation Plan # MES Alarm-Status API — Implementation Plan
**Date:** 2026-06-30 **Date:** 2026-06-30
**Status:** Draft plan — NOT yet executed. Captures design + task breakdown for review. **Status:** **Phase 1 [repo] COMPLETE 2026-08-01** — the `Alarms.CurrentAsync()` script accessor and the `AckTime` native-mirror enrichment are shipped (see §7). Phases 2–4 are **deployed config** (inbound methods + CvdReactor template scripts) and still pending; they need a live rig. All §6 open questions DECIDED 2026-08-01 (design review with user).
**Component touchpoints:** Inbound API (#14), Script Analysis (#25), Site Runtime (#3), Template Engine (#1) — plus deployed config (inbound methods + CvdReactor/MESReceiver template scripts). **Component touchpoints:** Inbound API (#14), Script Analysis (#25), Site Runtime (#3), Template Engine (#1) — plus deployed config (inbound methods + CvdReactor template scripts).
--- ---
@@ -12,9 +12,9 @@ Port the legacy **WWSupport / APIServer** MES alarm-status endpoints onto ScadaB
1. **Update** the existing `SimpleAlarmStatusRequest` inbound method (currently a stub) to do real work. 1. **Update** the existing `SimpleAlarmStatusRequest` inbound method (currently a stub) to do real work.
2. **Create** a new `AlarmStatus` inbound method (the full, filtered endpoint). 2. **Create** a new `AlarmStatus` inbound method (the full, filtered endpoint).
3. Both inbound methods **forward to MES-receiver site scripts** (`SAPID → BTDB machine lookup → Route.To(code).Call(...)`), exactly like `MesMoveIn`. 3. Both inbound methods **forward to site scripts** (`SAPID → BTDB machine lookup → Route.To(code).Call(...)`), exactly like `MesMoveIn`.
4. The **generic version on the `MESReceiver` template** resolves the machine's alarms by **querying the BTDB `MachineAlarm` table** (the legacy approach). 4. ~~The generic version on the `MESReceiver` template queries the BTDB `MachineAlarm` table.~~ **DROPPED (Q2, 2026-08-01):** no MESReceiver version ships — a config-only answer with no live state was judged not worth having. Machines whose template doesn't implement the scripts get a clean `WasSuccessful=false` "not supported on this machine" error from the inbound router.
5. The **`CvdReactor` template provides an override version** that reads the **alarms actually defined on the CvdReactor object** (its 7 native alarm-source bindings) directly — no `MachineAlarm` lookup. 5. The **`CvdReactor` template provides the (only) implementation**, reading the **alarms actually defined on the CvdReactor object** (its 7 native alarm-source bindings) directly — no `MachineAlarm` lookup anywhere.
Legacy spec being mirrored: [`docs/former-api-specs/mes/Alarm-API.md`](../former-api-specs/mes/Alarm-API.md). Legacy spec being mirrored: [`docs/former-api-specs/mes/Alarm-API.md`](../former-api-specs/mes/Alarm-API.md).
@@ -74,10 +74,11 @@ From the site-script API map:
``` ```
INBOUND (central) ENABLING CODE (repo) SITE TEMPLATE SCRIPTS (deployed config) INBOUND (central) ENABLING CODE (repo) SITE TEMPLATE SCRIPTS (deployed config)
───────────────── ──────────────────── ─────────────────────────────────────── ───────────────── ──────────────────── ───────────────────────────────────────
SimpleAlarmStatusRequest ─┐ Alarms script accessor ┌────► MESReceiver.SimpleAlarmStatus (BTDB MachineAlarm) SimpleAlarmStatusRequest ─┐ Alarms script accessor
├─ Route.To(code).Call(...) ──┤ MESReceiver.AlarmStatus (BTDB MachineAlarm) ├─ Route.To(code).Call(...) ─────────► CvdReactor.SimpleAlarmStatus (native sources)
AlarmStatus (new) ─┘ (ScriptRuntimeContext + └────► CvdReactor.SimpleAlarmStatus (native sources — OVERRIDE) AlarmStatus (new) ─┘ + AckTime mirror enrichment ──► CvdReactor.AlarmStatus (native sources)
ScriptCompileSurface) CvdReactor.AlarmStatus (native sources — OVERRIDE) (ScriptRuntimeContext +
ScriptCompileSurface) (no MESReceiver version — Q2 DECIDED: dropped)
``` ```
### 5.1 Layer A — Inbound API methods (deployed config; central) ### 5.1 Layer A — Inbound API methods (deployed config; central)
@@ -110,6 +111,8 @@ try {
> **Note:** Machine resolution stays on the inbound side (central) because it is environment/SQL-shaped and identical for both endpoints; site scripts receive an already-resolved `MachineCode` plus the alarm filter. > **Note:** Machine resolution stays on the inbound side (central) because it is environment/SQL-shaped and identical for both endpoints; site scripts receive an already-resolved `MachineCode` plus the alarm filter.
> **Unsupported machines (Q2 DECIDED):** since only CvdReactor implements the site scripts, `Route.Call` on any other machine fails script-not-found. Both routers catch that case specifically and return `{ WasSuccessful=false, ErrorText = $"Alarm status is not supported on machine '{code}'" }` instead of the raw exception text (verify during impl what exception/message `Route.Call` yields for a missing script so the catch can distinguish it).
### 5.2 Layer B — Enabling code change: script-facing `Alarms` accessor (repo) ### 5.2 Layer B — Enabling code change: script-facing `Alarms` accessor (repo)
New read-only accessor on the site script surface so scripts can enumerate the instance's current alarms. New read-only accessor on the site script surface so scripts can enumerate the instance's current alarms.
@@ -132,9 +135,14 @@ public sealed record ScriptAlarm(
string Message, string AlarmTypeName, string Category, string Message, string AlarmTypeName, string Category,
string OperatorUser, string OperatorComment, string OperatorUser, string OperatorComment,
DateTimeOffset? OriginalRaiseTime, DateTimeOffset Timestamp, DateTimeOffset? OriginalRaiseTime, DateTimeOffset Timestamp,
DateTimeOffset? AckTime, // Q4 DECIDED: real ack timestamp from the enriched mirror
string CurrentValue, string LimitValue, bool IsConfiguredPlaceholder); string CurrentValue, string LimitValue, bool IsConfiguredPlaceholder);
``` ```
**Ack-timestamp enrichment (Q4 DECIDED 2026-08-01 — enrich the mirror now, not a follow-up):**
- Add an additive `AckTime` (`DateTimeOffset?`) to `AlarmStateChanged`, the vendored `AlarmStateUpdate` proto (additive field number, never reuse — manual toggle-build-copy-untoggle regen), and the site `native_alarm_state` persistence so it survives failover.
- Semantics: when the underlying source supplies a true ack time (OPC UA A&C ack transitions do), use it; when it doesn't (MxGateway events without one), stamp the observation time of the ack transition at the DCL — accurate to when the system saw the ack, never fabricated. Null while unacked; cleared on re-raise.
Implementation: Implementation:
- Runtime: `Alarms.CurrentAsync()` Asks the Instance Actor for its alarm snapshot (reuse the existing internal alarm-state map that feeds `DebugViewSnapshot.AlarmStates`; project each `AlarmStateChanged` → `ScriptAlarm`). New internal request/response message (or reuse `DebugSnapshotRequest` and project off `AlarmStates`). - Runtime: `Alarms.CurrentAsync()` Asks the Instance Actor for its alarm snapshot (reuse the existing internal alarm-state map that feeds `DebugViewSnapshot.AlarmStates`; project each `AlarmStateChanged` → `ScriptAlarm`). New internal request/response message (or reuse `DebugSnapshotRequest` and project off `AlarmStates`).
- Compile surface: matching stub returning `Task.FromResult(empty)` so design-time compile + Test Run pass. - Compile surface: matching stub returning `Task.FromResult(empty)` so design-time compile + Test Run pass.
@@ -143,13 +151,9 @@ Implementation:
### 5.3 Layer C — Site template scripts (deployed config) ### 5.3 Layer C — Site template scripts (deployed config)
**MESReceiver (generic / "as before" — queries BTDB `MachineAlarm`):** `SimpleAlarmStatus` and `AlarmStatus`. **MESReceiver version — DROPPED (Q2 DECIDED 2026-08-01).** With no native alarm sources, MESReceiver has no live "triggered" signal; a configured-catalog-only answer was judged misleading rather than useful. No BTDB `MachineAlarm` read ships anywhere in this feature. Machines without the CvdReactor-style scripts get the router's "not supported on machine" error (§5.1).
- Use `await Database.Connection("BTDB")` (raw ADO.NET — site Database helper has **no** `QuerySingleAsync`; that's inbound-only) to read the machine's `MachineAlarm` rows (`Name`, `Severity`, `FlaggedForMES`, …) for the resolved `MachineCode`.
- Apply legacy filter semantics (Simple = flagged-only; full = `FlaggedOnly/MinSeverity/MaxSeverity/NameFilter`).
- Determine live "triggered" state + ack/timestamps and project to `AlarmInfo` (**live-state source = open question §6.2**).
- **No `_A`/`_B` side parsing.** This generic version is **whole-machine**, matching the legacy endpoint — it accepts a bare/numeric SAPID and never requires (or errors on) a missing side suffix. Side routing is **CvdReactor-only** (§5.3).
**CvdReactor (override — reads native sources directly, no DB):** `SimpleAlarmStatus` and `AlarmStatus` as **root-level scripts** (so `Route.To(code).Call("SimpleAlarmStatus")` resolves CvdReactor's version for CvdReactor instances — mirrors `MesMoveIn`). **CvdReactor (the only implementation — reads native sources directly, no DB):** `SimpleAlarmStatus` and `AlarmStatus` (Q5 DECIDED: names match the endpoints, template-agnostic contract) as **root-level scripts** (mirrors `MesMoveIn`).
The override **routes left vs right off the SAPID suffix** — `_A` ⇒ Left, `_B` ⇒ Right — and scopes the returned alarms to that side's native sources plus the shared reactor-wide source. The `_LT` (leak-test) suffix is **ignored** for scoping: it is stripped before reading the side, and both the side's run and leak-test sources are included. Source → side map (the 7 `CvdReactor` native sources): The override **routes left vs right off the SAPID suffix** — `_A` ⇒ Left, `_B` ⇒ Right — and scopes the returned alarms to that side's native sources plus the shared reactor-wide source. The `_LT` (leak-test) suffix is **ignored** for scoping: it is stripped before reading the side, and both the side's run and leak-test sources are included. Source → side map (the 7 `CvdReactor` native sources):
@@ -160,6 +164,10 @@ The override **routes left vs right off the SAPID suffix** — `_A` ⇒ Left, `_
```csharp ```csharp
// CvdReactor.SimpleAlarmStatus (sketch) // CvdReactor.SimpleAlarmStatus (sketch)
// Q1 DECIDED: MES relevance = dedicated severity band. Galaxy alarm priorities for
// MES-relevant alarms are configured into 900-999; nothing else may use that band.
const int MesBandMin = 900;
const int MesBandMax = 999;
try { try {
var raw = (Parameters["SAPID"] as string) ?? ""; var raw = (Parameters["SAPID"] as string) ?? "";
var code = Parameters["MachineCode"]?.ToString() ?? ""; var code = Parameters["MachineCode"]?.ToString() ?? "";
@@ -180,16 +188,17 @@ try {
var infos = alarms var infos = alarms
.Where(a => a.Active && !a.IsConfiguredPlaceholder) // only triggered .Where(a => a.Active && !a.IsConfiguredPlaceholder) // only triggered
.Where(a => InScope(a.NativeSourceCanonicalName)) // _A => Left*, _B => Right*, + ReactorAlarms .Where(a => InScope(a.NativeSourceCanonicalName)) // _A => Left*, _B => Right*, + ReactorAlarms
// SimpleAlarmStatus: flagged-only + acked always included (see §6.1 for FlaggedForMES) // SimpleAlarmStatus semantics: flagged-only (= MES band, Q1) + acked always included
.Where(a => a.Severity >= MesBandMin && a.Severity <= MesBandMax)
.Select(a => new { .Select(a => new {
Name = a.Name, Name = a.Name,
HierarchicalName = code + "." + a.Name, HierarchicalName = code + "." + a.Name,
Description = a.Message, // §6.3 Description = string.IsNullOrEmpty(a.Message) ? a.AlarmTypeName : a.Message, // Q3 DECIDED
IsFlaggedForMES = true, // §6.1 IsFlaggedForMES = a.Severity >= MesBandMin && a.Severity <= MesBandMax, // Q1 DECIDED: real predicate
Severity = a.Severity, Severity = a.Severity,
StatusCode = a.Acknowledged ? "Triggered.Acked" : "Triggered", StatusCode = a.Acknowledged ? "Triggered.Acked" : "Triggered",
TriggeredDT = (a.OriginalRaiseTime ?? a.Timestamp), TriggeredDT = (a.OriginalRaiseTime ?? a.Timestamp),
AckDT = (DateTime?)null, // §6.4 AckDT = a.AckTime, // Q4 DECIDED: enriched mirror
AckComment = a.OperatorComment, AckComment = a.OperatorComment,
}).ToList(); }).ToList();
return new { WasSuccessful = true, ErrorText = (string)null, Alarms = infos }; return new { WasSuccessful = true, ErrorText = (string)null, Alarms = infos };
@@ -197,29 +206,27 @@ try {
return new { WasSuccessful = false, ErrorText = "SimpleAlarmStatus failed: " + ex.Message, Alarms = Array.Empty<object>() }; return new { WasSuccessful = false, ErrorText = "SimpleAlarmStatus failed: " + ex.Message, Alarms = Array.Empty<object>() };
} }
``` ```
`CvdReactor.AlarmStatus` is the same — same side parsing + `InScope` filter — but additionally applies the passed `AlarmFilter` (NameFilter/MinSeverity/MaxSeverity/FlaggedOnly/IncludeAcked) over the scoped native list. (When `AlarmStatus` was selected by `Code`/`ZTag`/`MachineID` with no SAPID suffix, `side == null`; in that case return all sources instead of erroring — see §6.6.) `CvdReactor.AlarmStatus` is the same — same side parsing + `InScope` filter, WITHOUT the always-on band filter — but applies the passed `AlarmFilter` (NameFilter/MinSeverity/MaxSeverity/FlaggedOnly/IncludeAcked) over the scoped native list; `FlaggedOnly=true` means the MES-band predicate (Q1). `IsFlaggedForMES` is always reported per-row from the band predicate. (When `AlarmStatus` was selected by `Code`/`ZTag`/`MachineID` with no SAPID suffix, `side == null`; in that case return all sources instead of erroring — Q6b DECIDED.)
> **Shared `ReactorAlarms` (`Z28061.`) is included on both sides** — reactor-wide faults apply regardless of side. Flip this if MES wants strictly side-local alarms (drop the `StartsWith("Reactor")` clause). > **Shared `ReactorAlarms` (`Z28061.`) is included on both sides — Q6a DECIDED 2026-08-01:** reactor-wide faults apply regardless of which side MES asks about.
--- ---
## 6. Key design decisions & OPEN QUESTIONS (resolve before execution) ## 6. Key design decisions — ALL DECIDED 2026-08-01 (design review with user)
These are the Socratic checkpoints — they change script bodies and/or the contract. 1. **`IsFlaggedForMES` for native alarms — DECIDED: severity band.** MES relevance is encoded in the alarm severity itself: MES-relevant alarms are configured (in the Galaxy alarm priority) into a **dedicated band 900–999** reserved exclusively for MES-relevant alarms, so it cannot collide with ordinary criticality tuning. Scripts carry `MesBandMin = 900` / `MesBandMax = 999` as named constants; `IsFlaggedForMES = (Severity in band)` is reported honestly per row; `SimpleAlarmStatus` (always flagged-only) and `AlarmStatus` with `FlaggedOnly=true` filter by the band predicate. Since `Severity` is returned verbatim, MES's own `MinSeverity`/`MaxSeverity` filters compose naturally with the band. No new tables, code flags, or allow-lists. **Operational prerequisite:** the Galaxy alarm priorities for MES-relevant CvdReactor alarms must be set into 900–999 before the endpoints are meaningful.
1. **`IsFlaggedForMES` for native alarms.** The native model has no MES flag (it lived in `MachineAlarm.FlaggedForMES`). Options: (a) treat all native alarms as MES-relevant (`true`); (b) reintroduce an MES allow-list (per native source binding, or a small config/table) the override consults; (c) for `SimpleAlarmStatus` (flagged-only) return all, and only honor `FlaggedOnly` in the full endpoint by cross-referencing `MachineAlarm`. **Recommend (a)** for v1 with (b) as a follow-up, unless MES needs a true flag. 2. **MESReceiver version — DECIDED: dropped entirely.** With no native alarm sources there is no live state; a config-only catalog answer was judged not worth shipping. Only CvdReactor implements the scripts; other machines get the router's clean "not supported on machine" error (§5.1). No BTDB `MachineAlarm` dependency remains.
2. **MESReceiver live-state source.** MESReceiver has no native alarm sources, so after reading `MachineAlarm` config it has no live "InAlarm" source. Options: (a) MESReceiver version returns **configured alarms only** (no live triggered filter) — explicitly a degraded/reference path; (b) MESReceiver version ALSO reads `Alarms.CurrentAsync()` and cross-references `MachineAlarm` by name (works only if the instance happens to mirror native alarms); (c) declare the BTDB-only path returns config and document that live status requires the CvdReactor-style override. **Recommend (c)** — the real live path is CvdReactor; the MESReceiver version demonstrates the MachineAlarm-driven catalog and is the fallback for machine types catalogued in SQL. 3. **`Description` mapping — DECIDED:** `Message`, falling back to `AlarmTypeName` when `Message` is empty.
3. **`Description` mapping.** Legacy `Description` came from the MXAccess `DescAttrName` tag. `AlarmStateChanged` has no dedicated description; closest is `Message` (per-band operator text) / `AlarmTypeName` / `Category`. **Recommend `Message`, fall back to `AlarmTypeName`.** Confirm acceptable. 4. **`AckDT` mapping — DECIDED: enrich the native mirror now.** Additive `AckTime` on `AlarmStateChanged` + vendored proto + `native_alarm_state` (survives failover). True source ack time where available (OPC UA A&C); DCL observation time of the ack transition where the source lacks one (MxGateway); null while unacked; cleared on re-raise. See §5.2.
4. **`AckDT` mapping.** No explicit ack-timestamp in `AlarmStateChanged` (`Timestamp` = last change, `OriginalRaiseTime` = raise). For acked alarms we can't precisely fill `AckDT`. Options: (a) leave `null`; (b) use `Timestamp` when `Acknowledged` (approximate); (c) enrich the native mirror to carry an ack timestamp (larger change). **Recommend (a)** for v1, (c) as a follow-up if MES depends on it. 5. **Script naming — DECIDED: `SimpleAlarmStatus` / `AlarmStatus`** (match the endpoint names; template-agnostic contract any future machine template can implement). With the MESReceiver version dropped there is no parallel definition, so no override-resolution concern remains.
5. **Script naming / override resolution.** Keep script names **identical** across both templates (`SimpleAlarmStatus`, `AlarmStatus`) so the inbound method is template-agnostic (`Call("AlarmStatus")`), and CvdReactor's root-level definition is what `Route.Call` resolves for CvdReactor instances. (Alternative: `Mes`-prefixed CvdReactor scripts like `MesMoveIn`.) **Recommend identical names**; confirm `Route.Call` resolves a CvdReactor root script over any composed-module script of the same name (verify during impl — the MoveIn precedent says yes). 6. **Side scoping — DECIDED (all parts).** CvdReactor-only: `_A` ⇒ Left, `_B` ⇒ Right; `_LT` stripped before reading the side (leak-test sources still included). (a) shared `ReactorAlarms` (`Z28061.`) **included on both sides** — reactor-wide faults apply regardless of side. (b) missing suffix: `SimpleAlarmStatus` **errors** on a SAPID without `_A`/`_B` (matches `MesMoveIn`); `AlarmStatus` selected by `Code`/`ZTag`/`MachineID` (no SAPID) **returns all sources**; a SAPID selector without a suffix errors.
6. **Side scoping — DECIDED, CvdReactor-only.** Only the **CvdReactor** override routes off the SAPID suffix: `_A` ⇒ Left, `_B` ⇒ Right; `_LT` is ignored (stripped before reading the side; leak-test sources still included). Mirrors `MesMoveIn`. The generic **MESReceiver** version and the inbound method do **not** parse or require the suffix — they stay whole-machine (legacy). Two residual sub-choices remain: (a) **shared `ReactorAlarms` (`Z28061.`) included on both sides** (current plan) vs strictly side-local — confirm; (b) **missing `_A`/`_B` suffix** — the **CvdReactor** `SimpleAlarmStatus` **errors** (matches `MesMoveIn`); the **CvdReactor** `AlarmStatus` selected by `Code`/`ZTag`/`MachineID` (no SAPID) **returns all sources** rather than erroring; the MESReceiver version never errors on a missing suffix — confirm acceptable. 7. **Roles / auth — DECIDED:** no new roles; create `AlarmStatus` as Designer via CLI; authorize the **existing MES API key** (the one already calling `MesMoveIn`/`MesMoveOut`) for both alarm endpoints — one key per external system.
7. **New method roles / auth.** `AlarmStatus` is a new inbound `ApiMethod` (`X-API-Key`); creating it requires `Roles.Designer`. No new global roles. Confirm the API key in use is authorized.
--- ---
@@ -227,26 +234,39 @@ These are the Socratic checkpoints — they change script bodies and/or the cont
> Two artifact classes: **[repo]** = source/tests/docs committed to git; **[deployed]** = inbound methods / template scripts pushed to the cluster via CLI/UI (not in repo). The user said "don't execute yet" — this is the ordered plan only. > Two artifact classes: **[repo]** = source/tests/docs committed to git; **[deployed]** = inbound methods / template scripts pushed to the cluster via CLI/UI (not in repo). The user said "don't execute yet" — this is the ordered plan only.
**Phase 1 — Enabling `Alarms` script API [repo]** **Phase 1 — Enabling `Alarms` script API + AckTime mirror enrichment [repo]** — ✅ **DONE 2026-08-01**
1. Add `AlarmsAccessor` + `ScriptAlarm` to the runtime context (`ScriptRuntimeContext`) — local Ask to the Instance Actor; project `AlarmStateChanged` → `ScriptAlarm`. Add internal request/response message if not reusing `DebugSnapshotRequest`.
2. Mirror the stub on `ScriptCompileSurface` (and confirm `TriggerCompileSurface` not needed — trigger expressions don't read alarms). 1. ✅ **DONE 2026-08-01** — **AckTime enrichment (§6.4):** additive `AckTime` on `AlarmStateChanged`, the vendored `AlarmStateUpdate` proto (manual regen), `native_alarm_state` persistence, and the DCL ack-transition stamping (source ack time where supplied, else observation time).
3. Confirm/extend `ScriptTrustPolicy` allow-list so `Alarms` is permitted; no new forbidden APIs. - `AlarmStateChanged.AckTime` (init-only, `null` default) + `NativeAlarmTransition.AckTime` (trailing optional positional — all 14-arg call sites unchanged).
4. Unit tests: runtime accessor projection (active/acked/severity/timestamps), compile-surface compiles a representative `Alarms.CurrentAsync()` script, trust-policy accepts it. - Proto: **field 24** `google.protobuf.Timestamp ack_time` on `AlarmStateUpdate`; regenerated with `docker/regen-proto.sh sitestream` (csproj diff verified empty). Packed/unpacked by `StreamRelayActor` / `SiteStreamGrpcClient`; an absent Timestamp round-trips to `null`.
5. Doc: update `docs/requirements/Component-SiteRuntime.md` (+ Script Analysis #25 surface list) to document the `Alarms` accessor; note it in `Component-InboundAPI.md` routing examples. - Stamping rule (both protocols): non-null **only** while the condition is active AND acknowledged — that one predicate yields "null while unacked", "cleared on re-raise", and no phantom ack on the MxGateway return-to-normal (which maps `INACTIVE → Acknowledged = true`). Lives in the pure `OpcUaAlarmMapper.DeriveAckTime` / `MxGatewayAlarmMapper.DeriveAckTime`.
- OPC UA gets a **true source ack instant**: new SelectClause **index 18** = `AcknowledgeableConditionType/AckedState/TransitionTime`, appended so indices 0–17 keep their meaning; falls back to the event's `Time` when the server omits it. MxGateway uses the ack transition's own timestamp (its feed carries no ack time), and an `ACTIVE_ACKED` re-subscribe snapshot restores one from `LastTransitionTimestamp`.
- Persistence: rides `native_alarm_state`'s existing `metadata_json` blob, **not** a new column — that table is `RegisterReplicated` in `SiteLocalDbSetup` and LocalDb builds its CDC triggers from the column list at registration time, so an additive JSON property changes no schema, no triggers and no replication contract (`metadata_json` is exactly the extension point UA4 added). Pre-AckTime rows deserialize it as `null`.
2. ✅ **DONE 2026-08-01** — Add `AlarmsAccessor` + `ScriptAlarm` to the runtime context (`ScriptRuntimeContext`) — local Ask to the Instance Actor; project `AlarmStateChanged` → `ScriptAlarm` (incl. `AckTime`). Add internal request/response message if not reusing `DebugSnapshotRequest`.
- Dedicated `GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse` (Commons `Messages/Instance`) rather than reusing `DebugSnapshotRequest`, which would materialise every attribute value on every alarm poll. Served from the same `BuildAlarmStatesSnapshot()` the Debug View uses, so the two can never disagree.
- `AlarmsAccessor` sits in `ScopeAccessors.cs` beside the other accessors but is **not scope-prefixed** — alarm identity is not a scope-relative attribute name, so every scope sees the whole list. Exposed as `ScriptRuntimeContext.Alarms` and the top-level `ScriptGlobals.Alarms`.
3. ✅ **DONE 2026-08-01** — Mirror the stub on `ScriptCompileSurface` (and confirm `TriggerCompileSurface` not needed — trigger expressions don't read alarms).
- `CompileAlarmsAccessor` returns the **same** `ScriptAlarm` type as the runtime (Commons is already in `DefaultAssemblies`), so field access binds identically at the gate and at the site. `TriggerCompileSurface` confirmed not needed.
- Also mirrored on the **third** hand-maintained surface, the Central UI Test-Run `SandboxScriptHost` — without it the design page would false-flag CS1061 on scripts the deploy gate accepts. It throws a labelled `ScriptSandboxException` at run time (no central route to per-instance alarm state) rather than returning an empty list that would read as "nothing is in alarm".
4. ✅ **DONE 2026-08-01** — Confirm/extend `ScriptTrustPolicy` allow-list so `Alarms` is permitted; no new forbidden APIs.
- **No change needed, and the reason is structural:** the trust boundary is a deny-list over API roots, not an allow-list of context members. Pinned by a test asserting no `ForbiddenScopes` entry prefixes the Commons script-surface namespace, so a future deny-list entry cannot silently make `ScriptAlarm` untouchable.
5. ✅ **DONE 2026-08-01** — Unit tests: runtime accessor projection (active/acked/severity/timestamps/AckTime), AckTime stamping + failover persistence, compile-surface compiles a representative `Alarms.CurrentAsync()` script, trust-policy accepts it.
- New `AlarmsAccessorTests` (6), `NativeAlarmActor` AckTime emit/rehydrate/pre-AckTime-row (3), `InstanceActor` alarm-snapshot (2), mapper AckTime (4 OPC UA + 6 MxGateway), proto round-trip (1), Commons additive/back-compat (4), compile-surface + trust (4), `SandboxScriptHost` diagnose-clean (1). `AlarmsAccessor` added to the `CompileSurfaceParityTests` mirror pairs; the OPC UA SelectClause count lock-in went 18 → 19 with an index-18 assertion (intended — the clause is appended).
6. ✅ **DONE 2026-08-01** — Doc: update `Component-SiteRuntime.md` + `Component-DataConnectionLayer.md` (native-mirror AckTime) + Script Analysis #25 surface list; note the accessor in `Component-InboundAPI.md` routing examples.
- Also updated the `CLAUDE.md` native-alarm bullet. The Inbound API note records the *negative* decision: no `Route.To(...).GetAlarms(...)` verb — alarm reads go through a routed site script so the filtering happens where the data lives.
**Phase 2 — Inbound methods [deployed] + doc [repo]** **Phase 2 — Inbound methods [deployed] + doc [repo]**
6. Update `SimpleAlarmStatusRequest` (id 9) body to the §5.1 router (validate via design page first to avoid the stale-handler trap — see memory `inbound-noncompiling-update-keeps-old-handler`). 7. Update `SimpleAlarmStatusRequest` (id 9) body to the §5.1 router incl. the not-supported-machine catch (validate via design page first to avoid the stale-handler trap — see memory `inbound-noncompiling-update-keeps-old-handler`).
7. Create `AlarmStatus` method (params/return/script per §5.1). 8. Create `AlarmStatus` method (params/return/script per §5.1); authorize the existing MES API key for both methods (§6.7).
8. Doc the two endpoints in `docs/requirements/Component-InboundAPI.md` (or a dedicated MES-integration note) cross-referencing the legacy spec. 9. Doc the two endpoints in `docs/requirements/Component-InboundAPI.md` (or a dedicated MES-integration note) cross-referencing the legacy spec.
**Phase 3 — Site template scripts [deployed]** **Phase 3 — Site template scripts [deployed]** *(MESReceiver task removed — §6.2 decided dropped)*
9. Add `SimpleAlarmStatus` + `AlarmStatus` to **MESReceiver** (BTDB `MachineAlarm` via `Database.Connection("BTDB")`), per §6.2 decision. 10. Add `SimpleAlarmStatus` + `AlarmStatus` to **CvdReactor** (native sources via `Alarms.CurrentAsync()`, MES band constants), per §5.3.
10. Add override `SimpleAlarmStatus` + `AlarmStatus` to **CvdReactor** (native sources via `Alarms.CurrentAsync()`), per §5.3. 11. `template validate` (script compilation gate) before redeploy; redeploy affected instances.
11. `template validate` both templates (script compilation gate) before redeploy; redeploy affected instances.
**Phase 4 — Verify** **Phase 4 — Verify**
12. Build affected projects + run targeted tests (per memory `targeted-tests-not-full-suite`). 12. Build affected projects + run targeted tests (per memory `targeted-tests-not-full-suite`).
13. Live smoke against a real reactor instance (see §8) — both endpoints, success + machine-not-found + filtered. 13. Live smoke against a real reactor instance (see §8) — both endpoints, success + machine-not-found + not-supported-machine + filtered. Requires the Galaxy MES-band priorities (§6.1) to be set for a meaningful flagged-only result.
--- ---
@@ -261,10 +281,10 @@ These are the Socratic checkpoints — they change script bodies and/or the cont
## 9. Out of scope / follow-ups ## 9. Out of scope / follow-ups
- True `IsFlaggedForMES` for native alarms (per-source MES allow-list) — §6.1 option (b). - ~~True `IsFlaggedForMES` allow-list~~ — resolved by the §6.1 severity-band decision (no follow-up needed).
- Precise `AckDT` (enrich native mirror with an ack timestamp) — §6.4 option (c). - ~~Precise `AckDT` follow-up~~ — pulled INTO scope by §6.4 (mirror enriched now).
- A live/aggregated central alarm store or stream (the M7 follow-up) — these endpoints stay pull-based, per-instance, like the Alarm Summary page. - A live/aggregated central alarm store or stream (the M7 follow-up) — these endpoints stay pull-based, per-instance, like the Alarm Summary page.
- MESReceiver live-state path if §6.2 (c) is chosen and MES later needs live status from non-native machines. - Alarm-status support for non-CvdReactor machine types (any future template just implements `SimpleAlarmStatus`/`AlarmStatus` root scripts against the same contract).
--- ---
@@ -272,13 +292,14 @@ These are the Socratic checkpoints — they change script bodies and/or the cont
| Artifact | Type | Change | | Artifact | Type | Change |
|---|---|---| |---|---|---|
| `ScriptRuntimeContext` | [repo] | New `Alarms` accessor + `ScriptAlarm`; local alarm-snapshot Ask | | `AlarmStateChanged` + vendored `AlarmStateUpdate` proto (**field 24**) + `native_alarm_state` (`metadata_json`) + DCL stamping (OPC UA SelectClause **index 18**) | [repo] ✅ | Additive `AckTime` enrichment (§6.4) |
| `ScriptCompileSurface` | [repo] | Mirror `Alarms` stub | | `ScriptRuntimeContext` + `ScriptGlobals` | [repo] ✅ | New `Alarms` accessor + `ScriptAlarm` (incl. `AckTime`); local alarm-snapshot Ask |
| `ScriptTrustPolicy` (#25) | [repo] | Allow `Alarms` member (verify) | | `ScriptCompileSurface` + Central UI `SandboxScriptHost` | [repo] ✅ | Mirror `Alarms` stub on both design-time surfaces |
| Internal alarm-snapshot message (or reuse `DebugSnapshotRequest`) | [repo] | Additive | | `ScriptTrustPolicy` (#25) | [repo] ✅ | Verified — **no change needed** (deny-list, not member allow-list); pinned by test |
| Site Runtime + Script Analysis + Inbound API docs | [repo] | Document `Alarms` accessor + endpoints | | `GetAlarmSnapshotRequest`/`Response` (Commons) | [repo] ✅ | New, additive — chosen over reusing `DebugSnapshotRequest` |
| Unit tests (SiteRuntime / ScriptAnalysis / InboundAPI) | [repo] | New | | Site Runtime + DCL + Script Analysis + Inbound API docs + `CLAUDE.md` | [repo] ✅ | Document `Alarms` accessor, `AckTime`, endpoints |
| `SimpleAlarmStatusRequest` (id 9) | [deployed] | Stub → real router | | Unit tests (Commons / DCL / SiteRuntime / ScriptAnalysis / Communication / CentralUI) | [repo] ✅ | New |
| `AlarmStatus` (new) | [deployed] | New inbound method | | `SimpleAlarmStatusRequest` (id 9) | [deployed] | Stub → real router (+ not-supported catch) |
| `MESReceiver.SimpleAlarmStatus` / `.AlarmStatus` | [deployed] | New BTDB-driven scripts | | `AlarmStatus` (new) | [deployed] | New inbound method, existing MES key authorized |
| `CvdReactor.SimpleAlarmStatus` / `.AlarmStatus` | [deployed] | New native-source override scripts | | `CvdReactor.SimpleAlarmStatus` / `.AlarmStatus` | [deployed] | New native-source scripts (MES band 900–999) |
| Galaxy alarm priorities | [external] | MES-relevant CvdReactor alarms configured into 900–999 (§6.1 prerequisite) |
docs/plans/2026-07-08-deferred-work-register.md
+11 -5
View File
@@ -14,10 +14,14 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence | | 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling | | 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions | | 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 12 | Native-alarm-source-override CSV import — **Central UI `InstanceConfigure` upload affordance only** (CLI + Management API + parser shipped 2026-07-10, see Resolved) | m7 follow-ups | CLI/API path closes the operator parity gap; the Blazor upload button is polish | First request to bulk-import native sources from the UI rather than the CLI |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports | | 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 18 | Folder drag-drop | same, [PERM] | Permanently closed; menu reorder shipped | — (closed) |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs | | 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | Phase-8 WP-4 target-scale load test (10 sites × 500 instances × 75 tags = 37,500 subscriptions/site, 375,000 total) | `docs/plans/phase-8-production-readiness.md:152-170` (WP-4) + `:314-320` (test protocol); status claimed in `docs/plans/phase-8-checklist.md` | **Claimed complete but unevidenced.** The whole WP-4 deliverable is a **107-byte** checklist stub asserting "Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" with no per-work-package results and no linked run. Nearest real coverage is arithmetic/aggregation only — `PerformanceTests/StaggeredStartupTests.cs` (`TagCapacity_75TagsPer500Machines_37500Total`, 500-instances-over-10-sites distribution) and `HealthAggregationTests` (10-site report aggregation) — plus a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run exists anywhere in `tests/` or `docker/`. | Before any production go-live at target scale; or the first site approaching ~500 instances / ~37.5k subscriptions |
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
## Resolved (verified against the code 2026-07-10) ## Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability. Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
@@ -26,11 +30,12 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
|---|------|-----------| |---|------|-----------|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. | | 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. | | 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance still pending — see row 12 above.** | | 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance shipped 2026-08-01** — second `InputFile` on the `InstanceConfigure` Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (`InstanceConfigureNativeAlarmCsvImportTests`); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. | | 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox `Attributes.WaitAsync`/`WaitForAsync` (value-equality) route to the bound instance via `ISandboxInstanceGateway.WaitForAttributeAsync` → the existing `CommunicationService.RouteToWaitForAttributeAsync` cross-site route. Additive `RouteToWaitForAttributeRequest.RequireGoodQuality` (honored by the site handler) makes quality-gated waits route too. **Predicate-form waits stay unsupported** (an in-process lambda can't be routed) and throw a labelled `ScriptSandboxException`. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). | | 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox `Attributes.WaitAsync`/`WaitForAsync` (value-equality) route to the bound instance via `ISandboxInstanceGateway.WaitForAttributeAsync` → the existing `CommunicationService.RouteToWaitForAttributeAsync` cross-site route. Additive `RouteToWaitForAttributeRequest.RequireGoodQuality` (honored by the site handler) makes quality-gated waits route too. **Predicate-form waits stay unsupported** (an in-process lambda can't be routed) and throw a labelled `ScriptSandboxException`. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): `RealOpcUaClient` sets `Truncated=false`/`ContinuationToken=null` on the last page; `BrowseNodeResult` carries both; `TreeRow.razor` renders "Load more" only when a continuation token remains — no wasted BrowseNext. | | 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): `RealOpcUaClient` sets `Truncated=false`/`ContinuationToken=null` on the last page; `BrowseNodeResult` carries both; `TreeRow.razor` renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: `StubOpcUaClient` supports browse + address-space search, covered by `StubOpcUaClientBrowseTests`/`StubOpcUaClientSearchTests`. | | 16 | StubOpcUaClient throws on browse | Already resolved: `StubOpcUaClient` supports browse + address-space search, covered by `StubOpcUaClientBrowseTests`/`StubOpcUaClientSearchTests`. |
| 18 | Folder drag-drop | Closed — **permanently deferred (`[PERM]`)** by the M9 decision (`docs/plans/2026-06-15-stillpending-completion-design.md:122`): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (`2026-05-11-templates-folder-hierarchy-design.md:27`). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): `PendingDeploymentPurgeActor` central singleton (spawned in `AkkaHostedService`) ticks `IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync` every `CommunicationOptions.PendingDeploymentPurgeInterval` (default 1h), options-validated, tested. | | 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): `PendingDeploymentPurgeActor` central singleton (spawned in `AkkaHostedService`) ticks `IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync` every `CommunicationOptions.PendingDeploymentPurgeInterval` (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. | | 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (`docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8): new `KpiRollupHourly` table (migration `20260710153953`) folded by a third recorder tick (`kpi-rollup`, `RollupInterval` default 1h) over a re-folded `RollupLookbackHours` window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (`KpiMetricAggregationCatalog`); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by `RollupThresholdHours` (default 168h); longer rollup retention (`RollupRetentionDays` default 365 ≥ `RetentionDays`, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (`Component-KpiHistory.md`), and tests shipped. | | 22 | KPI history hourly rollups | Shipped 2026-07-10 (`docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8): new `KpiRollupHourly` table (migration `20260710153953`) folded by a third recorder tick (`kpi-rollup`, `RollupInterval` default 1h) over a re-folded `RollupLookbackHours` window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (`KpiMetricAggregationCatalog`); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by `RollupThresholdHours` (default 168h); longer rollup retention (`RollupRetentionDays` default 365 ≥ `RetentionDays`, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (`Component-KpiHistory.md`), and tests shipped. |
@@ -41,12 +46,13 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator | | Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components | | docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component | | Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | **Trigger fired 2026-07-08** — the "PLAN-01 rig landing" trigger fired when `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs` landed (before the placeholder harness even shipped), and went unnoticed (NF2). **Owner: PLAN-R2-01** — `archreview/plans/PLAN-R2-01-*.md` (failover-envelope measurement task) owns wiring `FailoverTimingTests` to the fixture rig or recording the true blocker. | Owned by PLAN-R2-01; this register row tracks the handoff only | | ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
| Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure** (the still-open half of the former combined row) | Never measured, and no owner plan survives now that PLAN-R2-01 closed the failover half. `PerformanceTests` covers failover timing, staggered startup, health aggregation, audit hot-path latency and a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs` — nothing measures store-and-forward drain throughput, nor what a slow/stalled subscriber does to the per-subscriber buffering in `Communication/Actors/StreamRelayActor.cs` / `Grpc/SiteStreamGrpcServer.cs` with many subscribers attached. No defect observed; deferred as measurement-only work. | First field S&F backlog that fails to drain within an operator's patience, a slow gRPC subscriber degrading a site stream for others, or the WP-4 target-scale run (row 25) being scheduled — that run should absorb this |
## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12) ## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence. Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
| # | Item | Where noted | Rationale for deferral | Revisit trigger | | # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------| |---|------|-------------|------------------------|-----------------|
| SBR | **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition *without* the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until the crashed node restarts; only a younger-node crash fails over. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` | Remedy is a production SBR topology/strategy decision (keep-majority + a 3rd/lighthouse seed node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. | Before the next production deployment that adds a central node, or the first real active-node crash. | | SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** | | vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |
docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md
+4 -1
View File
@@ -1,6 +1,9 @@
# Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10) # Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10)
**Status:** Draft plan (not yet executed) — 2026-07-10 **Status:** **Delivered 2026-07-10** — shipped as designed (`SiteAlarmAggregatorActor`,
`ISiteAlarmLiveCache`, the `SubscribeSite` streaming RPC, and
`AlarmSummaryService.BuildFromLiveAlarms` are all in `src/`). The 15 s poll is retained as the
NotReporting authority behind the live stream, per the design below.
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10 **Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10
**Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."* **Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."*
**Owning component:** Central UI (#9) + Central–Site Communication (#5); no new component. **Owning component:** Central UI (#9) + Central–Site Communication (#5); no new component.
docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md
+4 -1
View File
@@ -1,6 +1,9 @@
# Plan: KPI History Hourly Rollups (deferred #22) # Plan: KPI History Hourly Rollups (deferred #22)
**Status:** Draft plan (not yet executed) — 2026-07-10 **Status:** **Delivered 2026-07-10** — shipped as designed: the `KpiRollupHourly` entity +
EF migration `20260710153953_AddKpiRollupHourlyTable`, `KpiMetricAggregationCatalog`
(per-metric gauge-vs-rate fold), and the raw-vs-rollup query routing on `RollupThresholdHours`
are all in `src/`.
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #22 **Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #22
**Revisit trigger that fired this plan:** *"KpiSample query latency on dashboards."* **Revisit trigger that fired this plan:** *"KpiSample query latency on dashboards."*
**Owning component:** KPI History (#26); touches Configuration Database (#17), **Owning component:** KPI History (#26); touches Configuration Database (#17),
docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json
+1 -1
View File
@@ -191,5 +191,5 @@
} }
], ],
"lastUpdated": "2026-07-19", "lastUpdated": "2026-07-19",
"phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. Branch feat/localdb-phase1 NOT merged/pushed." "phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. MERGED to main as 28ca04d7 ('LocalDb adoption Phase 1 + 2: consolidate the site database, delete the bespoke replicators (#23)') - Phase 1 and Phase 2 landed together in that one merge."
} }

Some files were not shown because too many files have changed in this diff Show More