dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dohertj2/ScadaBridge:feat/localdb-phase1
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0
..
pull from: dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:ui/central-ui-cleanup-sweep dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0

163 Commits
feat/localdb-phase1 .. ui/central-ui-cleanup-sweep

Author SHA1 Message Date
Joseph Doherty d14e0ee4b1 fix(ui): gate detail modals on user intent, not on the row resolving 
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).

The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.

Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.

Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.

The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
 2026-08-11 06:03:06 -04:00
Joseph Doherty 9e243493fb ui: Central UI density/consistency sweep + Theme 0.4.1 
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.

The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.

Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.

NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.

Build 0/0; suite green apart from that one intermittent failure.
 2026-08-11 05:50:12 -04:00
Joseph Doherty b6f383a225 docs(notifications): record EWS live-gate PASS (rig -> on-prem Exchange, Delivered first attempt) 2026-08-10 07:29:06 -04:00
Joseph Doherty ba994a59c5 docs: final-review nits — test-stub wording, README tech-stack row, UI-audit follow-up 2026-08-10 07:07:00 -04:00
Joseph Doherty 05d0631cdd chore(plans): mark EWS transport plan tasks 10-11 complete 2026-08-10 07:00:12 -04:00
Joseph Doherty 00d8a923af docs(notifications): EWS transport docs; close Q12 as superseded; design-doc corrections from execution reviews 2026-08-10 06:56:56 -04:00
Joseph Doherty 82f52e81ce fix(ui): mirror the EWS username:password credentials rule on the SMTP page 2026-08-10 06:53:15 -04:00
Joseph Doherty 3e490f2a7d test(ui): bUnit coverage for the SMTP page transport selector state machine 2026-08-10 06:50:39 -04:00
Joseph Doherty acbb9eafa5 fix(management): EWS write gate requires username:password credentials 2026-08-10 06:48:15 -04:00
Joseph Doherty 8df15b34b8 fix(transport): carry Transport + OAuth2 authority/scope on SmtpConfigDto (OAuth2 fields were silently dropped) 2026-08-10 06:43:43 -04:00
Joseph Doherty 2f217f5742 docs(cli): document notification smtp update --transport 2026-08-10 06:39:49 -04:00
Joseph Doherty 6f8b9c755d feat(ui): EWS transport selector on /notifications/smtp 2026-08-10 06:38:46 -04:00
Joseph Doherty 0fe1972960 feat(cli): notification smtp update --transport smtp|ews 2026-08-10 06:35:33 -04:00
Joseph Doherty 4e633b1e64 feat(management): Transport on UpdateSmtpConfigCommand with EWS shape validation 2026-08-10 06:33:29 -04:00
Joseph Doherty 8657fae14f fix(notifications): EWS sender review follow-ups — https guard, transient-path logging, test hardening 2026-08-10 06:29:37 -04:00
Joseph Doherty 08957cc907 feat(notifications): EWS branch in the email delivery adapter 2026-08-10 06:27:19 -04:00
Joseph Doherty 2ee5586406 test(notifications): correct XXE guard comment — LINQ-to-XML does not prohibit DTDs by default 2026-08-10 06:21:19 -04:00
Joseph Doherty 6864890e5f fix(notifications): prohibit DTD processing in EwsResponseParser (XXE guard) 
The review asked for a regression test pinning DTD-prohibited parsing. Writing
it showed the premise was wrong: XDocument.Parse permits an internal DTD subset
and expands its entities, so a DOCTYPE-bearing response body parsed fine and the
guard did not exist (entity-expansion DoS on external input).

Parse now goes through XmlReader with DtdProcessing.Prohibit and a null
XmlResolver. The regression test feeds a DOCTYPE + ENTITY payload shaped as a
well-formed EWS error response, so it fails if DtdProcessing is ever loosened
rather than passing for the unrelated-XML reason.
 2026-08-10 06:20:48 -04:00
Joseph Doherty abc58e6394 feat(notifications): no-SDK EWS SOAP mail sender with typed transient/permanent classification 2026-08-10 06:18:08 -04:00
Joseph Doherty abb1581d45 feat(notifications): EWS CreateItem envelope builder + response parser 2026-08-10 06:12:46 -04:00
Joseph Doherty 3da84825fc feat(notifications): additive Transport column on SmtpConfigurations 2026-08-10 06:09:12 -04:00
Joseph Doherty d21b7a5a48 feat(notifications): EmailTransport enum + parser for the EWS transport discriminator 2026-08-10 06:07:14 -04:00
Joseph Doherty 9f4d7d4bcb docs(notifications): EWS email transport implementation plan (11 tasks) 2026-08-10 06:05:04 -04:00
Joseph Doherty b2ea9c6c74 docs(notifications): design EWS email transport for the outbox; close Q12 as superseded 
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
 2026-08-10 05:56:41 -04:00
Joseph Doherty e9c412e528 fix(host): unhandled boot exception now kills the process instead of wedging the container (#34) 
Root cause: dotnet runs as container PID 1 and Linux ignores default-action
signals sent to PID 1, so the runtime's unhandled-exception path (banner,
then abort() -> SIGABRT) could never terminate the process — it printed the
trace and spun the main thread at 100% CPU with the container `running`,
so `restart: unless-stopped` never fired. Reproduced deterministically:
same StartupValidator throw exits 134 under an init process and wedges
without one.

Two layers, each covering the other's gap:
- Program.cs registers an AppDomain.UnhandledException handler before the
  first statement that can throw: prints the trace, best-effort flushes
  Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform,
  134 preserves the 128+SIGABRT crash code, and it covers every thread,
  not just the boot window. It cannot fire under WebApplicationFactory
  (the test host catches entry-point exceptions), so the designed
  boot-refusal exceptions still propagate to tests unchanged.
- docker-compose: init: true on all 8 nodes for the crash paths that
  bypass the managed event (Environment.FailFast, runtime-internal aborts).

The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out
(exactly one call, only inside the handler); Environment.Exit still fires
the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the
crash path skips nothing abort() kept. New pin test keeps the handler ahead
of the configuration build.

Live-verified on the rig image: crash now yields Exited (134) +
RestartCount climbing under `unless-stopped`, trace intact, with and
without init; full 8-node rig redeployed healthy with docker-init as PID 1.

Closes #34.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-08 05:23:26 -04:00
Joseph Doherty e697477c1f feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator 
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 11:39:28 -04:00
Joseph Doherty 4d7f09d550 test(secrets): central-shared-store live gate 5/5 — shared SQL store + hub failover proven on the rig 
Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 11:28:58 -04:00
Joseph Doherty 68f812eaa4 feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.0 and wire hub fallback endpoints 
Pin all five ZB.MOM.WW.Secrets* packages 0.4.1 -> 0.5.0, which brings
SecretsGrpcHubClientOptions.FallbackEndpoints and the package's internal
FailoverSecretsHubReader. A site whose GrpcHub section lists fallback
endpoints now fails a sweep over to the next central instead of stalling
on a downed primary - safe ONLY because both central nodes serve one
shared SQL secret store (scadaproj#4), so either hub answers with the
same manifest; the appsettings comments say so and warn against listing
endpoints backed by independent stores.

appsettings.json gains "FallbackEndpoints": [] with a _fallbackEndpoints
comment, and the _endpoint note's single-endpoint-stall caveat is scoped
to the empty-list case it now only applies to.

Wiring pins (red first on 0.4.1): site + Grpc + one fallback resolves
ISecretsHubReader to FailoverSecretsHubReader with the
"zb-secrets-grpc-hub:fallback:0" keyed channel present; zero fallbacks
keeps the plain GrpcSecretsHubClient and no fallback channel - the
pre-0.5.0 container shape byte-identical.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:55:38 -04:00
Joseph Doherty aebdd56b52 docs(secrets): review follow-ups — pre-Serilog window note, ThrowIfNull hygiene, honest site-purity scan bound 
The expander comment now records that a SQL outage at central boot exits
pre-Serilog with a bare stderr trace (honest, restart-retryable, unenriched)
and why the role-reading predicate deliberately stays in Program.cs rather
than SecretsRegistration (that class refuses config-read roles by design).
EnsureCentralSharedStoreConnectionString gains the file's standard
ThrowIfNull. The site-purity scan's doc no longer overclaims: two
factory-lambda descriptors evade it individually; the pin holds because
three concrete-type registrations from the same call cannot.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:47:10 -04:00
Joseph Doherty a358244d9e fix(secrets): share the central connstr validation with the Layer-A expander 
The blank/${secret:} pre-checks lived only in AddScadaBridgeSecrets, but on
a real central boot the Layer-A expander runs FIRST and would hand a bad
value to SqlConnection, burying the designed message under a generic
'initialization string' format error. Extracted both checks into
EnsureCentralSharedStoreConnectionString — one definition, called by the
expander (earliest point) and by registration (covers embedded/test
composition) — same single-source lesson as the UsesGrpcHub predicate.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:36:13 -04:00
Joseph Doherty 43e87a7492 feat(secrets): central's Grpc-mode store is the SHARED SQL-Server store (scadaproj#4) 
In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.

- SecretsRegistration: two fail-closed pre-checks before any registration on
  the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
  naming the key (an independent store per central node is the recorded
  defect), and a value containing ${secret: throws naming the bootstrap
  circularity (the expander needs this store to resolve references). Site
  registrations are byte-identical to before; SqlServer mode and
  replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
  non-blank connection string migrates and resolves pre-host ${secret:}
  references through the shared SQL store, so expanded values can never
  diverge from what the running node serves. Every other case keeps the
  SQLite path unchanged; blank-connstr central deliberately falls through so
  the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
  central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
  ${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
  descriptor sweep), central fixtures carry the now-required connstr.

Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 10:33:28 -04:00
Joseph Doherty f6c3f7c593 test(secrets): live gate 4/4 — check 4 re-run and PASSES on 0.4.1 
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.

The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.

Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.

Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.

The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.

Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.

4/4. Merging.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:51:39 -04:00
Joseph Doherty 6d38e89be0 deps(secrets): 0.4.1 — hub denial warning (live-gate check 4) 
The 2026-08-07 live gate's check 4 failed one clause of three: both auth
negatives were denied with a byte-identical Unauthenticated status and no token
or plaintext reached any log, but the hub recorded a denial only at Information
level, via Grpc.AspNetCore.Server rather than the gate itself. A follower whose
token was mis-rotated would therefore stop converging while central showed
nothing above INF.

That was a property of the library, not of this branch, so it was fixed there and
shipped as 0.4.1 (scadaproj main c86cead): SecretsHubAuthInterceptor now emits a
rate-limited Warning summarising each denial window, breaking the count down by
cause, with the first denial after startup or a quiet window warning immediately
so a single probe is never silent. Wire behaviour is untouched - the denial is
still one uniform Unauthenticated with one detail, and still carries no token
material - so this is additive on the server's diagnostics only and nothing a
follower observes changes.

All five ZB.MOM.WW.Secrets* pins move together. Splitting them is not an option
worth having: Abstractions carries the StoredSecret shape the Grpc wire mirror is
written against, so a mixed set is a silent structural mismatch rather than a
build error.

Build clean at 0 warnings; the secrets wiring + hub-mapping pins are 31/31.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:36:57 -04:00
Joseph Doherty 9d5cf7100e test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged 
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:01:55 -04:00
Joseph Doherty fc784b4137 fix(secrets): registration and hub mapping share the UsesGrpcHub predicate (review Important) 
AddScadaBridgeSecrets now branches on UsesGrpcHub instead of re-deriving the
same condition, so the doc's single-predicate claim is enforced rather than
aspirational; the role split is a switch with a default-throw so a future
third role must choose its hub half explicitly.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:33:47 -04:00
Joseph Doherty c8e90daafb test(secrets): pin both hub halves, the mode key, and the mapping gate 
Extends the existing build-and-resolve wiring suite rather than asserting over
ServiceDescriptors, for the reason that file already documents: a registration
can look correct as a descriptor list and still fail on first resolve.

Role separation is pinned in both directions — central registers the hub's
fail-closed interceptor and NOT the sweep, a site registers the sweep and NOT
the hub — because only one of those is a security failure and testing the
happy half would not catch it. The sweep is asserted by resolving IHostedService,
which builds its whole graph (reader, keyed channel, local store) and is where
a forgotten AddZbSecrets would surface; GrpcSecretSyncService is internal to
the package, so it is matched by assembly + type name the way the SqlServer
replicator's services already are.

SqlServer mode gets regression pins with the mode key both unset and named
explicitly, plus one asserting it stays role-agnostic — both nodes sync
bidirectionally against the same database, and the role parameter added for the
hub must not have quietly changed that.

The mapping tests assert over the app's real endpoint data sources, on the WIRE
route (/zb.mom.ww.secrets.hub.v1.SecretsHub/...) rather than the C# type, since
the route is what a follower addresses. One of them pins exactly the two READ
methods: pull-only is a property of the contract, and this is where a future
package version growing a write RPC would become visible instead of silently
opening a path for a site to overwrite central.

Numeric mode values get their own test. Enum.TryParse accepts any integer,
including ones outside the enum, so "7" would otherwise select a mode that does
not exist and fall through to the SqlServer branch.

Verified red-first by mutation on the finished implementation: swapping the two
role branches reds 8 (both role pins, both fail-closed pins, both mapping
pins); deleting the UsesGrpcHub check in the map extension reds exactly the two
"maps no hub endpoint" cases — the unauthenticated-hub scenario; dropping
Enum.IsDefined and letting UsesGrpcHub ignore Enabled reds the out-of-range
value and the flag-off-with-full-hub-config cases. 31/31 green restored.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 127ec25425 feat(secrets): wire the pull-only gRPC secrets hub — central hosts, sites sweep 
scadaproj#3: the production secrets topology is central hosting a pull-only
gRPC hub with site nodes sweeping it. The SqlServer replicator stays in the
codebase and keeps working exactly as it did, but it is not the production
path — it needs every site node to hold a connection string to central's
database, which breaks ScadaBridge's standing rule that sites talk to central,
not to central's DB.

Selection is a new Secrets:Replication:Mode key alongside the existing
Secrets:Replication:Enabled flag. Absent or blank means SqlServer, so an
existing configuration that sets only Enabled behaves identically; an
unrecognised value is refused at startup naming the key, and refused whenever
it is present rather than only when replication is on — a typo should fail the
boot that introduced it, not some later boot that flips an unrelated flag.

Which HALF a node composes is a parameter, not a config key. Both composition
roots already know statically which they are (Program.cs is central,
SiteServiceRegistration is a site), and a role read from configuration is a
role that can be got wrong in the one direction that matters: a site hosting
the hub would serve central's whole secret inventory from inside the site
network to anything holding the shared token.

The hub is mapped onto the EXISTING central h2c control-plane listener
(ScadaBridge:Node:CentralGrpcPort, default 8083) beside CentralControlService
— the same listener and the same addressing convention sites already use, and
the same shape as the site's LocalDb sync endpoint sharing its gRPC port: two
disjoint service prefixes, two independent fail-closed gates. The
CentralControlAuthInterceptor on AddGrpc is prefix-scoped and passes hub calls
through; the hub's own SecretsHubAuthInterceptor, attached per-service by
AddZbSecretsGrpcHub, gates them on Secrets:GrpcHub:BearerToken.

Registration and mapping share one predicate (UsesGrpcHub) deliberately.
Mapping without registering would map a hub whose interceptor was never
attached — an anonymous endpoint serving every secret central holds, on a node
that looks completely healthy — so the two must not be able to drift.

gRPC mode fails CLOSED where SqlServer mode degrades: a missing endpoint or
bearer token is a startup failure, not a warning plus a local-only store. The
degraded outcome is precisely what the hub exists to prevent (a site quietly
serving secrets that never converge), and the package's own errors name the
exact key, so they are left unwrapped.

Templates are default-OFF: Enabled stays false, Mode stays SqlServer, and
Secrets:GrpcHub ships with an empty BearerToken and Endpoint. Empty is
fail-closed, not open. The token is documented as appsettings/env only — never
a ${secret:} reference, since resolving one is what the hub exists to make
possible — the same bootstrap rule the mesh pre-shared keys follow.

Known asymmetry, recorded in the template: CentralGrpcEndpoints is a LIST that
fails over across the central pair, but the hub client dials a SINGLE endpoint,
so a sweep against a downed central node stalls instead of failing over. That
is survivable — the sweep is best-effort and the site keeps serving its full
local last-known-good store — but secrets stop converging until that node is
back.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 57f08c213b deps(secrets): bump ZB.MOM.WW.Secrets* to 0.4.0 and take the Grpc replicator 
0.4.0 adds ZB.MOM.WW.Secrets.Replicator.Grpc — the pull-only central secrets
hub scadaproj#3 selected as ScadaBridge's production topology. The four
existing pins move with it rather than straddling two versions: they share the
Abstractions surface the replicators bind to, and a mixed set is a restore that
resolves but composes types from two different builds of the same seam.

One package carries both halves (server + sweep client); which half a node
composes is a registration-time decision, so only the Host — the composition
root for both roles — references it.

nuget.config already maps ZB.MOM.WW.Secrets.* to the dohertj2-gitea feed, so
no source-mapping change was needed. Restore verified against the feed.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:17 -04:00
Joseph Doherty 4df3a55824 docs: truth sweep — retire stale registers, reconcile ledgers with shipped state 
- deferred.md: DELETED (git rm) — stale 2026-07-10 duplicate of the canonical
  deferred-work register; this completes archreview R2-08 T11 (the file was
  tracked, not untracked as the task assumed)
- ScadaBridge-docs-issues.md, ScadaBridge-docs-fixed.md: DELETED (git rm) —
  generated 2026-07-10 CommentChecker reports, already consumed; completes
  R2-08 T13 (also tracked, not untracked)
- stillpending.md: prepended historical-snapshot banner (2026-06-15 audit;
  Tier-1 table is not current open work)
- docs/plans/phase-8-checklist.md: replaced the unevidenced 'Complete / All
  passing' stub with the honest state per register row 25 (WP-4 target-scale
  load test never run)
- archreview/plans/00-MASTER-TRACKER.md: R2-01 T2 live failover drill
  annotated RESOLVED 2026-08-01 (PLAN-R2-01 T4 + docker/failover-drill.sh +
  SbrFailoverTests); R2-08 T11/T13 recorded completed by this sweep
- docs/plans/2026-07-22-clusterclient-to-grpc-plan.md: P3 deferred-RPCs note
  updated (all four live-proven 2026-08-01, 1c99d6fa); ClusterClientSiteAuditClient
  naming follow-up marked DONE (63c16d69)
- docs/plans/2026-05-28-opcua-tag-browser.md.tasks.json: Task 19 flipped to
  completed (manual smoke PASS 2026-08-01, 6dc5d94c)
- archreview/plans/PLAN-R2-0[1-8]*.tasks.json: all-pending manifests reconciled
  with the authoritative tracker (round 2 merged @ 1930f19b) — flipped to
  completed except R2-08 T1/T2 which remain pending needs-user
- docs/operations/2026-07-16-secrets-clustered-master-key.md: correction banner
  (SQL-hub replication shipped 8e12f994; KEK-rotation + clustered-secrets
  runbooks ship with ZB.MOM.WW.Secrets)
- docs/plans/2026-07-19-localdb-phase2-live-gate.md: external-system-delete
  observation annotated RESOLVED (2d03f2d5 reconciles deletions incl.
  external_systems)

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 01:56:33 -04:00
Joseph Doherty 7caa8bfd99 fix(localdb): disable SQLite pooling on legacy reads - a Windows site node could not boot 
SiteLocalDbLegacyMigrator opens each pre-Phase-2 file read-only, copies its rows into
the consolidated site database, then renames the file so a later boot skips it.
Microsoft.Data.Sqlite POOLS connections, so disposing one returns it to the pool and
leaves the underlying sqlite3 handle - and the OS file handle - open. On Windows the
subsequent File.Move throws IOException ("being used by another process"), the
migration faults out of AddZbLocalDb's factory, and the site node does not start at
all. There is no partial-migration path: it is a hard boot failure, once, on the first
upgrade past LocalDb Phase 2.

Why it shipped: POSIX rename ignores open handles, so this cannot reproduce on Linux or
macOS. Every existing rename assertion in SiteLocalDbLegacyMigratorTests - including
LegacyTrackingRows_AreCopiedAndTheFileIsRenamed - passes with the bug fully present, and
the docker rig migrates cleanly. It was found by pre-flighting the wonder-app-vd03
upgrade against COPIES of that box's real site databases; an earlier pre-flight pass
with empty data directories had nothing to drain and passed clean.

Fix: both legacy read-only connection strings now go through one
LegacyReadOnlyConnectionString helper carrying Pooling=False.

The accompanying test is deliberately white-box. A behavioural assertion cannot
discriminate here on the platform this suite runs on, so it pins the connection string
instead; it is red without the fix.

Verified: Host.Tests 440/440, 0 warnings. The build deployed to wonder-app-vd03 carries
this change (as the then-uncommitted fix) and has been running since 2026-08-05.
 2026-08-05 17:06:54 -04:00
Joseph Doherty b8f91bab2d feat(cluster): enable the #33 bootstrap guard on the docker rig — live gate PASS 
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
 2026-08-02 01:04:55 -04:00
Joseph Doherty 3c9b101dfc docs(alarms): document the Alarms accessor + AckTime, tick MES plan Phase 1 
Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:

  - Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
    it is not scope-prefixed, why it is read-only, why placeholder rows are
    included), the full ScriptAlarm shape, AckTime on the enriched
    AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
    persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
    builds its CDC triggers from the column list at registration time).
  - Component-DataConnectionLayer.md already carried the AckTime section in the
    first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
  - Component-ScriptAnalysis.md: accessors returning domain types return the
    SAME type on both surfaces, and the trust-model note that a deny-list needs
    no entry for a new globals member -- only that its return type resolves in
    a permitted namespace.
  - Component-InboundAPI.md records the NEGATIVE decision: there is
    deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
    per-instance and lives on the site's Instance Actor, so the read goes
    through a routed site script and the filtering happens where the data is;
    central stays a thin router.
  - CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
  - The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
    what was actually built (incl. the two choices that differ from the plan's
    "or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
    and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
    stay open -- they are deployed config and need a live rig.
 2026-08-01 13:12:46 -04:00
Joseph Doherty d0af884760 feat(scripts): add the Alarms.CurrentAsync() read accessor for site scripts 
MES alarm-status API §5.2 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 tasks 2-4). Site `Call` scripts had NO way to read alarm condition
state: the `Alarm` global exists only inside an on-trigger handler and
describes the one alarm that fired, and native mirrored conditions were
reachable only from the Debug View. That gap blocked the CvdReactor
SimpleAlarmStatus/AlarmStatus scripts entirely -- they cannot be written
without it. `Alarms.CurrentAsync()` closes it.

The data was already local: the script runs inside its own Instance Actor's
context, so this is a LOCAL Ask -- the same mechanism attribute reads use, no
cross-cluster hop. A dedicated GetAlarmSnapshotRequest/Response is used rather
than reusing DebugSnapshotRequest, which would materialise every attribute
value on every alarm poll; both are served from the same
BuildAlarmStatesSnapshot(), so the script view and the operator's Debug View
can never disagree.

Deliberate shape decisions:
  - NOT scope-prefixed, unlike Attributes. Alarm identity is not a
    scope-relative attribute name (computed alarms are keyed by configured
    name, native conditions by a source-supplied reference), so prefixing
    would hand a composed script a silently truncated list.
  - Read-only. Native alarms are a read-only mirror of the source (no
    ack-back), so no acknowledge/shelve operation is exposed.
  - Placeholder rows are NOT pre-filtered: a caller must be able to tell
    "binding configured and quiet" from "binding unknown". The documented
    filter is `Active && !IsConfiguredPlaceholder`.
  - ScriptAlarm lives in Commons so the runtime accessor and the compile-only
    surface project to the SAME type -- a script binding at the design-time
    gate binds identically at the site. Condition is the authority for
    active/acked/severity, so one filter expression works across computed and
    native alarms.

Mirrored on BOTH design-time surfaces. ScriptCompileSurface is covered by the
reflection parity guard (AlarmsAccessor added to its mirror pairs). The Central
UI Test-Run SandboxScriptHost is the third, hand-maintained mirror that the
parity test cannot reach (Central UI does not reference Site Runtime); without
it the design page would false-flag CS1061 on scripts the deploy gate accepts.
It throws a labelled ScriptSandboxException at run time -- there is no central
route to per-instance alarm state, and returning an empty list would read as
"nothing is in alarm", which is worse than an error.

ScriptTrustPolicy needs NO change, and the reason is structural rather than
incidental: the trust boundary is a deny-list over API roots, not an allow-list
of context members. A test pins that no ForbiddenScopes entry prefixes the
Commons script-surface namespace, so a future deny-list entry cannot silently
make ScriptAlarm untouchable.

Tests: 6 accessor cases (Ask contract, full native projection incl. AckTime,
unacked, computed-alarm derivation, placeholder visibility, scope-independence),
2 InstanceActor snapshot cases incl. equality with the Debug View row set, the
full MES script shape compiling against ScriptCompileSurface, 2 trust cases,
and a sandbox diagnose-clean case reading every projected ScriptAlarm field.
 2026-08-01 13:12:30 -04:00
Joseph Doherty 01bcca992c feat(alarms): thread an additive AckTime through the native-alarm mirror 
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.

Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.

Provenance is honest, never fabricated:
  - OPC UA A&C supplies a TRUE ack instant, so we now select it:
    AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
    index 18, APPENDED so the positional reads at 0-17 keep their meaning.
    Servers that omit the field fall back to the event's own Time.
  - MxAccess Gateway supplies none, so the ack transition's own timestamp is
    used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
    re-subscribe snapshot restores one from LastTransitionTimestamp rather
    than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.

Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.

Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.

Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
 2026-08-01 13:12:04 -04:00
Joseph Doherty 6dc5d94cb9 docs(plans): tag-browser Task 19 manual smoke PASS 2026-08-01 — online + offline paths verified live 2026-08-01 12:41:24 -04:00
Joseph Doherty 819ca4d7ce docs(plans): retire the folder-hierarchy manual smoke — drag-drop steps obsolete under the [PERM] decision, rest covered by bUnit suites + CLI parity 2026-08-01 12:37:48 -04:00
Joseph Doherty a949fe6f57 docs(plans): retire the env2 + transport manual checklists — core scenario run live 2026-08-01 (caught + fixed the create-missing FK bug), rest covered by automated suites + #31 2026-08-01 12:34:34 -04:00
Joseph Doherty 0c9dffed78 fix(transport): flush created-site ids before the connection pass — create-missing import failed FK 547 on real SQL Server 
Caught live 2026-08-01 importing a bundle into the empty env2 cluster: a
create-missing Site has Id == 0 until SaveChanges on a relational provider,
and ApplyDataConnectionsAsync stamps DataConnection.SiteId as a raw scalar
(no navigation, no EF fix-up), so the insert violated
FK_DataConnections_Sites_SiteId. Every importer integration test runs on
the EF in-memory provider, which assigns ids eagerly on AddAsync — the
exact masking the in-code comment predicted. Fix: one SaveChangesAsync
between the site pass and the connection pass, riding the same outer
transaction (all-or-nothing preserved; the failed live import rolled back
cleanly). Regression test runs the create-missing path on SQLite
(CreateMissingSiteRelationalTests) — red without the fix, green with it.
 2026-08-01 12:32:45 -04:00
Joseph Doherty 1c99d6fa8d docs(grpc-plan): live gate fully closed — ExecuteOpcUa/ExecuteRoute/parked-retry/TriggerSiteFailover all live-proven 2026-08-01 2026-08-01 12:25:27 -04:00
Joseph Doherty 9ab27d5d61 docs(register): rows 27-28 — ES retry config never reaches sites (found live), failover dialog mislabeled Delete 2026-08-01 12:02:23 -04:00
Joseph Doherty f6822f8f45 docs(m10): close follow-up #163 — InstanceConfigureListOverrideTests verified green (#207 re-verified too) 2026-08-01 11:28:46 -04:00
Joseph Doherty 410349767d style(centralui): full-app bg-light/bg-white -> theme-aware utility sweep (M10 residual) 
T34c only fixed the bounded modal-surface offenders. Bootstrap 5.3's bg-light
and bg-white are fixed light values that do NOT flip under [data-bs-theme=dark],
so every remaining use was a dark-mode contrast break. 35 swaps across 19 files:

  surface / <pre> / <code>  bg-light                -> bg-body-secondary
  panel                     bg-white                -> bg-body
  neutral badge             bg-light text-dark      -> bg-secondary-subtle text-secondary-emphasis
  muted badge / input group bg-light text-muted     -> bg-body-secondary text-body-secondary

DELIBERATELY LEFT (7 sites): the neutral member of a status-badge switch or
ternary whose siblings are all solid, non-theme-aware colours (bg-success,
bg-danger, bg-warning) — swapping only the neutral one to a subtle token breaks
the visual weight of the set, so these stay until the whole family is retoned:
  Topology.razor:513 (Current badge) and :588 (InstanceState.NotDeployed)
  InstanceConfigure.razor:1520 (same InstanceState switch)
  NotificationReport.razor StatusBadgeClass fallback
  TransportImport.razor ConflictKind badge fallback
  SecuredWrites.razor text-bg-light fallback (a text-bg-* family with no
    theme-aware member at all)
  Health.razor:377 depth ternary
Also untouched by design: SchemaBuilder.razor:88 (already bg-light-subtle,
theme-aware), bg-dark text-light console panels, and site.css / #reconnect-modal.

InstanceConfigure.razor is edited concurrently elsewhere; its change here is 5
pure class-string swaps on existing lines, no reflow. No test asserted any of
these classes. Also marks all four M10 residuals done in the plan doc.
 2026-08-01 11:28:20 -04:00
Joseph Doherty 316153dc98 feat(centralui): hide OffsetPager on a single page, opt-in — NotificationReport enabled (M10 residual) 
DECISION: a pager whose only two controls are permanently disabled is noise, so
the conventional behaviour ships — the bar is hidden when the result set is a
single page. NotificationReport, which raised the finding, opts in.

RATIONALE for opt-in rather than default-on. OffsetPager is shared, and its
summary span ("Page N of Y · T total") doubles as the ONLY total-count readout
on ConfigurationAuditLog — auto-hiding there would silently delete the result
count for every query returning under a page, and the existing Playwright
fixture asserts "3 total" on exactly such a query. So the behaviour lives in the
shared component (reusable for the next consumer) behind HideWhenSinglePage,
default false, which keeps every current consumer byte-identical.

The guard fires only when single-page-ness is POSITIVELY established:
PageCount <= 1 AND Page <= 1 AND !HasNextPage. A null TotalCount means the host
cannot count, so the bar renders — the controls never vanish while a further
page still exists.

NotificationReport's own Playwright pagination test seeds 51 rows (2 pages) and
is unaffected. 5 new bUnit tests cover hide-on-single-page, hide-on-empty,
show-on-multi-page, show-when-TotalCount-unknown, and the default-off path.
 2026-08-01 11:28:20 -04:00
Joseph Doherty a506b19d17 refactor(centralui): migrate TemplateEdit's four page-embedded modals to the DialogService host (M10 residual) 
TemplateEdit was the last page still hand-rolling its own modal chrome after
M10 — T34c only tokenized its backdrops. All four member-authoring forms
(Attribute, Alarm, Native Alarm Source, Script) now open through
IDialogService.ShowAsync, so the single DialogHost in MainLayout owns the
backdrop, focus trap, Escape and focus restoration.

Pattern copied from the already-migrated pages (MoveDataConnectionDialog +
Templates' Move/Rename dialogs): each body is its own component beside the page
taking a DialogContext<bool>, owning its form state, rendering validation and
server errors INLINE while staying open, and closing with Close(true) only once
the save succeeded — at which point the page reloads. An inline RenderFragment
would NOT have worked: DialogHost renders the captured fragment in its own tree,
so the page's StateHasChanged could never refresh it.

Persistence deliberately stayed on the page (it owns TemplateService, the
inherited-member rules, and the repository-direct native-source path) and is
reached through an OnSaveAsync delegate returning null on success or the message
to display. Behaviour preserved verbatim, including the List-attribute encode +
Decode round-trip check, the name/trigger-type read-only-on-edit rules, the
duplicate-native-source-name guard, and NormalizeExecutionTimeout.

TemplateScriptDialog keeps all four tab panels mounted (Monaco and the JSONJoy
island must not tear down on tab switch) and hosts the Test Run panel, which
needs the live unsaved editor buffer, so it injects ScriptAnalysisService
directly and cancels an in-flight run on dispose.

Extraction moved markup that three structural source-scanning tests pinned;
all three were repointed at the new files rather than weakened:
  - TemplateNativeAlarmSourceEditorTests (+ a new test asserting the form is
    host-mounted and the body renders no chrome of its own)
  - AttributeListEditorTests (list-editor reveal now in the dialog body)
  - TestRunWarningTests (Real I/O warning travelled with the script panel)

Build 0/0; CentralUI.Tests 973/973 green.
 2026-08-01 11:28:20 -04:00
Joseph Doherty fdfd5e1b27 feat(centralui): TreeView full WAI-ARIA keyboard navigation (M10 residual R7) 
TreeView<TItem> handled only Enter/Space on the chevron; the tree itself was
unreachable by keyboard. Implements the WAI-ARIA tree pattern:

- Roving tabindex on li[role=treeitem] — exactly one node is tabbable, so the
  whole tree is a single Tab stop. Target resolved per render: last-focused node
  while still visible -> SelectedKey -> first visible node. Browser focus follows
  via a per-node ElementReference + FocusAsync in OnAfterRenderAsync, guarded by
  the same JSException/JSDisconnectedException/InvalidOperationException triple
  the context-menu focus already used (a no-op under bUnit).
- ArrowDown/ArrowUp move between VISIBLE nodes; ArrowRight expands a collapsed
  branch else moves to first child; ArrowLeft collapses an expanded branch else
  moves to parent; Home/End jump to first/last; Enter/Space activate through the
  SAME path a click takes (OnContentClick / OnCheckboxToggle) so selection
  semantics never diverge between input modes.
- ARIA: aria-level, aria-posinset, aria-setsize added; aria-selected now renders
  true/false on selectable+checkbox trees and stays absent on non-selectable ones.
- Event scoping: @onkeydown:stopPropagation on the li, chevron, checkbox and
  content slot, so nested nodes do not double-handle and consumer controls inside
  NodeContent keep their own key handling. Browser scroll-on-Space/Arrow is
  suppressed by a minimal NATIVE inline onkeydown on the root ul, targeted at
  treeitems only — Blazor's preventDefault directive is all-or-nothing per
  element, so on the li it would trap Tab and cancel Enter/Space on consumer
  buttons. No CSP is configured, so the inline handler runs.

BuildVisibleNodes() mirrors RenderNode's visibility rules and must stay in step.

Tests: 31 new bUnit tests in TreeViewKeyboardNavigationTests; the 47 existing
TreeView tests are unchanged and still green. Docs: new keyboard/a11y section in
docs/components/TreeView.md.
 2026-08-01 11:28:20 -04:00
Joseph Doherty 8aa6bf2270 fix(audit): populate ParentExecutionId on alarm-triggered script runs 
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.

Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:

  ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
    → SetStaticAttributeCommand.SourceExecutionId
    → AttributeValueChanged.SourceExecutionId   (InstanceActor static-write path)
    → AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext

All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.

Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).

Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.

Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
 2026-08-01 11:22:07 -04:00
Joseph Doherty 88638d774a feat(cli,management): close area-move and template-folder CLI parity gaps 
Two verified-absent parity gaps between the service layer and the CLI /
ManagementActor command surface, both left as follow-ups by the 2026-05-11
design plans.

(1) area move. AreaService.MoveAreaAsync had existed since the deployment
topology page shipped but was reachable only from the Blazor UI. Adds
MoveAreaCommand(AreaId, NewParentAreaId?) to Commons, a ManagementActor
dispatch arm delegating straight to AreaService.MoveAreaAsync (not-found /
self-parent / descendant-cycle / cross-site / name-collision all surface as
the standard curated ManagementCommandException failure response; the service
writes its own "Move" audit row), and the CLI verb `site area move --id
[--parent-id]`. Omitting --parent-id moves the area to the site root, matching
the command's nullable NewParentAreaId. The command carries the SAME any-of
[Designer, Deployer] gate as CreateArea/UpdateArea/DeleteArea (arch-review C6):
re-parenting is the same structural authoring act, exposed on the same two
surfaces. Placed under the existing `site area` group rather than a new
top-level `area` group, alongside its create/update/delete siblings.

(2) template folder verbs. The five folder management commands have been
handled by ManagementActor since the folder-hierarchy plan, but the promised
CLI surface was never written. Adds `template folder
list|create|rename|move|reorder|delete` mapping 1:1 onto ListTemplateFolders /
CreateTemplateFolder / RenameTemplateFolder / MoveTemplateFolder /
ReorderTemplateFolder / DeleteTemplateFolder. --parent-id is omitted to target
the tree root; --direction takes the lowercase literals up/down, validated at
parse time by AcceptOnlyFromAmong (same case-sensitive contract as the audit
--channel/--kind/--status options).

Follow-on updates: the frozen authorization matrix gains its MoveArea entry
(reflection-driven, so a missing entry would have failed CI); CommandTreeTests
pins both new verb sets plus the omit-parent-id-means-root parse behaviour and
registry round-trips; ManagementActorTests covers the MoveArea role gate and
the delegate-to-service success/root/cycle/not-found paths; the CLI README and
Component-ManagementService.md document the new surface (the latter also gained
the previously-undocumented ReorderTemplateFolder); both plan docs' follow-up
lines are marked done.
 2026-08-01 11:15:24 -04:00
Joseph Doherty e0851e3e17 docs(register): close row 12 — InstanceConfigure native-alarm-source CSV upload shipped 2026-08-01 11:12:10 -04:00
Joseph Doherty a26d6ba317 feat(central-ui): native-alarm-source CSV import on InstanceConfigure 
The Native Alarm Source Overrides card had no bulk affordance — the CSV path
shipped CLI-only (instance native-alarm-source import --file), so the UI could
only retarget one source at a time inline.

Adds a second InputFile on that card, mirroring the attribute importer's UX
(hidden input behind a button-styled label, 512 KB cap, success/error alert
with the per-line error list, toast). Parsing reuses the SHARED
NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the
new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same
batch rules the server enforces in
ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must
resolve, must not be template-locked, and may appear at most once; any error
rejects the whole file and applies nothing.

Semantics match the CLI: merge, not full replace — sources absent from the
file keep their existing override, a blank field keeps the inherited value,
and an all-blank row clears that source's override (equivalent to the CLI's
all-null override row, without leaving a dead row behind).

Persistence reuses the inline editor's path: SaveNativeOverride's upsert body
is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the
import commits the validated batch in one SaveChangesAsync — no new server
method, no duplicated parsing.

Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field
inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source
and parser-error rejection, plus structural pins on the InputFile wiring.
Docs: Component-CentralUI.md native-alarm-source card gains the import bullet.
 2026-08-01 11:11:40 -04:00
Joseph Doherty 697be0ce43 docs(topology): fix stale keep-oldest reference in Site Cluster Behavior — clusters run auto-down since 2026-07-21 2026-08-01 11:10:48 -04:00
Joseph Doherty 4558bc3b1f docs(register): track 4 untracked deferrals, split the failover/perf row, close #18 
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).

Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.

Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
 2026-08-01 11:10:19 -04:00
Joseph Doherty 2d03f2d507 fix(site-runtime): reconcile artifact deletions on apply — central deletes no longer orphan site rows 
The artifact apply (DeploymentManagerActor.HandleDeployArtifacts) was
upsert-only, so deleting an external system (or shared script, DB connection,
data connection) centrally never removed the site's SQLite row — a deleted
external system stayed callable from site scripts forever. Central always
ships the COMPLETE set of each artifact class (ArtifactDeploymentService
GetAll* snapshots; the wire's presence-tracking wrapper lists preserve
null-vs-empty), so the site now applies upsert-then-reconcile: after storing
the incoming set, SiteStorageService.DeleteRowsExceptAsync removes any stored
row absent from it, per artifact table. A null list still means 'field not
shipped' and touches nothing.

Runtime cleanup rides along: a reconciled-away shared script is unregistered
from the compiled SharedScriptLibrary (a stale delegate would stay callable
until restart), and a removed data connection is evicted from the DCL hash
cache and its live connection actor stopped via the previously-caller-less
RemoveConnectionCommand — both on the actor thread via the extended
ApplyArtifactDataConnectionsToDcl message. All four tables are
RegisterReplicated, so the deletes reach the standby as ordinary CDC row
tombstones.

Tests: storage-level reconcile per table (incl. empty-set-deletes-all and
idempotency) in ArtifactStorageTests; actor-level pins in
DeploymentManagerActorTests (orphan delete, null-set no-op, library
unregistration, DCL stop for the removed connection only). Docs:
Component-DeploymentManager + Component-SiteRuntime record the
full-set/reconcile semantics.
 2026-08-01 10:54:18 -04:00
Joseph Doherty 0123b68719 docs(plans): MES alarm-status API — all 7 open design questions decided 
Design review 2026-08-01: (1) MES relevance = dedicated severity band 900-999
(script constants, real IsFlaggedForMES predicate); (2) MESReceiver version
DROPPED — CvdReactor is the only implementation, router returns a clean
not-supported error elsewhere; (3) Description = Message else AlarmTypeName;
(4) enrich the native mirror NOW with an additive AckTime (proto +
native_alarm_state + DCL stamping); (5) script names match the endpoints;
(6) shared ReactorAlarms on both sides, suffix behavior as planned;
(7) existing MES API key, no new roles. Plan is now ready to execute.
 2026-08-01 10:29:44 -04:00
Joseph Doherty d1ade5653b docs(plans): bookkeeping sync — reconcile stale trackers with merged code 
A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:

- Flip ~380 stale task statuses to completed across March/May/June/July
  trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
  LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
  code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
  HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
  template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
  (premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
  -> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
  scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
  merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
  sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
  TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
  #28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
  seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
  requirements-traceability 'Pending' clarified as frozen plan-generation
  status.

Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.
 2026-08-01 08:53:56 -04:00
Joseph Doherty 663d03e89c Merge branch 'fix/cached-telemetry-drain-hot-loop' 2026-07-27 15:50:42 -04:00
Joseph Doherty 63c16d6912 refactor(comm): retire ClusterClient naming after the gRPC cutover 
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.

Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.

Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
 2026-07-27 15:40:01 -04:00
Joseph Doherty b3dc17a5fe Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0) 2026-07-24 13:35:27 -04:00
Joseph Doherty 1c9159a7ce fix(tests): qualify Health page type — Communication.Health namespace shadowed it 2026-07-24 13:35:24 -04:00
Joseph Doherty 5d4853a1f0 refactor(health): adopt the shared active-node check (Health 0.3.0) 
OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck
selected by cluster leadership (review 01 [High]): leadership is address-ordered
and diverges from singleton placement after a restart, and during a partition
both sides compute themselves leader so Traefik served both. Health 0.3.0 makes
the shared check age-based — it is now this host's own rule, promoted — so the
private copy is deleted and central registers the shared type.

ActiveNodeEvaluator, the "THE single definition of active node" this repo already
maintained, now delegates to the shared ClusterActiveNode rather than
re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp,
the inbound-API gate and the /health/active tier on one rule, and it means the
rule is shared with OtOpcUa, which had independently written a third copy.
Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering
trade-off is recorded at the PackageReference.

SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the
rule — it is a thin adapter over the site's own IClusterNodeProvider, which
scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering
the shared check directly would have to re-derive the site role and would lose the
property that the tier and singleton placement come from the same provider.
Central stays unscoped: every central member competes for one active slot.

No behaviour change on any node — same rule, one implementation instead of two.

Verified: Host.Tests 439/439, Communication ActiveNode 2/2.
 2026-07-24 13:21:58 -04:00
Joseph Doherty c1216044a4 docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings 
Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the
bootstrap guard' subsection (dark switch, founder/probe semantics, config table,
accepted trade) plus a forward reference from Dual-Node Recovery. Host
appsettings.Central/Site.json carry a disabled BootstrapGuard block with a
_bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover
note added.
 2026-07-24 08:57:44 -04:00
Joseph Doherty 248676ed16 feat(cluster): simultaneous-cold-start split-brain guard (#33) 
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
 2026-07-24 08:55:48 -04:00
Joseph Doherty 47850c0f53 feat(health): serve MapZbHealth on site nodes 
Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and
/metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site
node whether it was ready or which half of the pair was active. The family
overview dashboard probes every instance the same way, and this is the one gap.

Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so
the composition-root tests that build this graph actually cover them) and mapped
by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside
/metrics:

  akka-cluster  [Ready]  the shared AkkaClusterHealthCheck. Also carries the
                         cluster-view data (leader/memberCount/...) the dashboard
                         reads, free with ZB.MOM.WW.Health 0.2.0.
  localdb       [Ready]  NEW SiteLocalDbHealthCheck — SELECT 1 through the
                         registered ILocalDb. Site has no EF context; central's
                         DatabaseHealthCheck<ScadaBridgeDbContext> is central-only.
                         Replication state rides along as data ENRICHMENT only:
                         replication is default-OFF, so failing on it would mark
                         every correctly-configured node unready.
  active-node   [Active] NEW SitePairActiveNodeHealthCheck, delegating to
                         IClusterNodeProvider.SelfIsPrimary. Deliberately NOT
                         central's OldestNodeActiveHealthCheck: that one calls
                         SelfIsOldest(cluster) with no role argument and would
                         compute "oldest" across the wrong member set on a mesh
                         carrying more than one site.

Anonymous, as central's are: the site pipeline runs no authentication middleware
and has no FallbackPolicy, so nothing extra was needed.

Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free).

Tests: SiteHealthCheckTests builds the REAL site container and activates every
registration through its factory (exact-set names, one tier tag each, resolved
types, central-only checks absent behind a positive control) plus behaviour for
both new checks. SiteHealthEndpointTests boots the real site Program over
WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting
MapZbHealth would leave every registration test green while site nodes 404'd.

Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.
 2026-07-24 05:54:30 -04:00
Joseph Doherty bb138d5254 docs: add env_variables.md — full environment-variable inventory 
Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI,
DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra
containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a
shipped appsettings key already exists for it, consumer, purpose, potential
values, and required-ness. Compiled from GetEnvironmentVariable call sites, the
Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1,
and all appsettings*.json.
 2026-07-24 05:02:24 -04:00
Joseph Doherty 9a12826172 docs(plans): OtOpcUa v3 native-alarm B/C live-gate PASS (#14) — no code needed; item C premise obsolete (Gitea #17) 2026-07-24 04:33:20 -04:00
Joseph Doherty 86d129de6c fix(dcl): harden endpoint host-rewrite — IPv6 brackets + portless URLs 
Code review of a1abbff7 found two edge cases: a discovery/advertised URL with no
explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an
IPv6 literal host could lose/double its brackets. Omit the port when neither URL
carries one; bracket an IPv6 host only when missing. +4 tests (12 total).
 2026-07-23 16:58:44 -04:00
Joseph Doherty dbec3ee263 docs(plans): OtOpcUa v3 raw-path live-gate PASS (#14) — fix validated live, B/C deferred 2026-07-23 16:54:48 -04:00
Joseph Doherty a1abbff760 fix(dcl): rewrite advertised OPC UA endpoint host to the reachable one 
An OPC UA server advertises its base address from its own config, not the route
the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT
hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl
verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and
the connect failed. RealOpcUaClient now swaps the advertised authority for the
reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path;
no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).
 2026-07-23 16:43:45 -04:00
Joseph Doherty 266f001a2e docs(plans): OtOpcUa v3 dual-namespace cutover scope + phase-2 plan (#14) 2026-07-23 15:22:22 -04:00
Joseph Doherty e04c2617cc feat(ui): warn on bare ns= node bindings in the browse picker (v3 cutover #14) 2026-07-23 15:18:36 -04:00
Joseph Doherty 97afa84fcd docs(dcl): sweep ns= examples to durable nsu= form (v3 cutover #14) 2026-07-23 15:17:13 -04:00
Joseph Doherty 0eb44314cb feat(dcl): add OpcUaReferenceForm.IsDurable — flags bare ns= bindings (v3 cutover #14) 2026-07-23 15:16:47 -04:00
Joseph Doherty a5256e9b12 chore(comm): delete dead IntegrationCallRequest routing (Gitea #32) 
'Pattern 4: Integration Routing' — RouteIntegrationCallAsync →
SiteEnvelope(IntegrationCallRequest) → SiteCommunicationActor integration
handler — was plumbed end to end but connected at neither end: no
producer (zero callers) and no handler (AkkaHostedService never
registered LocalHandlerType.Integration). It was an early scaffold the
architecture routed around — the brokered External→Central→Site→Central
round-trip is served by the Inbound API's routed-site-script path (the
RouteTo* verbs, driven from CommunicationServiceInstanceRouter), which is
live, tested, and shares IntegrationTimeout.

Decision (#32): delete. Removed the IntegrationCall{Request,Response}
messages, RouteIntegrationCallAsync, the SiteCommunicationActor receive
block + _integrationHandler field + LocalHandlerType.Integration, and the
four tests that covered them (2 actor, 2 message-contract, 1 dispatcher-
reject, 1 mapper-reject). KEPT IntegrationTimeout — it is the live
timeout for the RouteTo* verbs. Updated the exclusion-narrative comments
(proto/mapper/dispatcher), design §4, the components doc timeout table,
and marked the known-issue RESOLVED.

Full solution build clean (0/0); Communication 634 + Host 421 green.
Net -172/+20 across 14 files. Not in the gRPC proto (was deliberately
excluded there), so no wire-format change.
 2026-07-23 14:27:32 -04:00
Joseph Doherty e0f105c3b3 docs(env2): live-gate docker-env2 on the gRPC PSK build — PASS 3/3 (Gitea #31) 
Rebuilt scadabridge:latest from main @ 8524a7f7 and recreated only the
env2 containers. All three gate checks pass on site-x:
1. both site nodes boot with the key (StartupValidator fail-closed →
   reaching 'Application started' proves the key present);
2. control-plane PSK auth: no-header / wrong-key ⇒ PermissionDenied,
   correct key ⇒ success, on both nodes (:9123, :9124);
3. LocalDb unaffected (local-only; 0 errors, healthy boot).
Bonus: central registers site-x online via gRPC heartbeat; no real
ClusterClient/receptionist (only the benign ClusterClientSiteAuditClient
label, same as the primary rig). Noted a seed-data gap (ScadaBridgeConfig2
dbo.Sites is empty) — orthogonal to the transport.
 2026-07-23 14:10:22 -04:00
Joseph Doherty 8524a7f746 test(sms-e2e): valid Twilio SID fixture so the create path passes (Gitea #29) 
TestAccountSid was 'ACtest123' (AC + 6 chars). The management create
path validates the Account SID against ^AC[0-9a-fA-F]{32}$
(ManagementActor.cs:2205, added 2026-07-10, commit 40088a21) while the
fixture predates the guard (2026-06-19) — so the create was rejected and
the config-card + secret-non-leak assertions have been red/inert since.

Use AC + 32 hex (AC00000000000000000000000000000001) so the create
succeeds and the downstream Auth-Token-non-leak assertion actually runs
again. Unit-level RepositoryCoverageTests keep their short SIDs — they
construct SmsConfiguration directly and never hit the validated path.
 2026-07-23 14:03:49 -04:00
Joseph Doherty c4dcd9bc02 fix(transport): run bundle import inside the execution strategy (Gitea #28) 
BundleImporter.ApplyAsync opened a user-initiated transaction directly,
but the central ConfigurationDb context is configured with
EnableRetryOnFailure. SqlServerRetryingExecutionStrategy rejects
user-initiated transactions, so every bundle import threw on any real
SQL Server ('...does not support user-initiated transactions.'). The
whole Transport suite ran on the in-memory provider (no retrying
strategy, BeginTransaction is a no-op) so it never surfaced.

Extract the transactional apply into ApplyMergeAsync and drive it via
_dbContext.Database.CreateExecutionStrategy().ExecuteAsync, so the
strategy owns the BeginTransaction -> apply -> Commit unit as one
retriable block. The delegate resets per-attempt state (change tracker
+ ApplyMergeAsync rebuilds its own summary/resolution accumulators) so a
retried attempt cannot double-apply; a rollback failure is captured and
surfaced on the BundleImportFailed audit row exactly as before. Post-
commit side effects (ScriptArtifactsChanged publish, session zero/
remove) moved outside the retriable delegate so they run once.

Regression test: BundleImporterRetryingStrategyTests runs the import on
SQLite with a retrying execution strategy (RetriesOnFailure == true,
arming the same guard as production). It fails with the exact production
error on the pre-fix code and passes after. Existing rollback/apply
contracts unchanged (104 integration + 154 unit tests green).
 2026-07-23 13:55:09 -04:00
Joseph Doherty 9f91e84d83 docs(grpc): Phase 5 live gate — PASS (8/8), migration complete 
Full eight-check gate on the Phase 4 deletion build (main @ 7fd5cb2b),
rig rebuilt from main. All 8 checks PASS: PSK negatives; site->central
matrix (notif no-loss/dupe, both audit paths, reconcile self-heal);
central->site matrix (Query/Parked/Lifecycle); active-central kill ->
sticky flip 1s + central-b active 26s; mid-drain total==distinct;
297,574-byte gRPC reply (128KB frame-class retired); cluster membership
pair-only (no cross-boundary Akka association); full-rig restart 0
receptionist/ClusterClient lines on any of 8 nodes.

Check-1 clarification recorded: site-side interceptor uses the node's
single GrpcPsk and ignores x-scadabridge-site (a central-side routing
hint) -- per-site isolation proven by wrong-key reject. Instance-
dependent central->site RPCs (OpcUa/Route/standby-parked/TriggerFailover)
carried forward, unit-proven.
 2026-07-23 13:35:16 -04:00
Joseph Doherty 7fd5cb2b56 feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only 
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.

Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
  ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
  AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
  coexistence flags; the CentralTransportMode/SiteTransportKind enums

gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.

Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).

Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.

Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.

Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
 2026-07-23 12:54:32 -04:00
Joseph Doherty 3a8ddb7087 docs(known-issues): link IntegrationCallRequest dead-code note to Gitea #32 2026-07-23 11:52:52 -04:00
Joseph Doherty 54da10dc00 docs(grpc): Phase 3 live gate — PASS; central→site command cutover 
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.

Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
  (SoakNotify instances survived the recreate as Enabled)

Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
  TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
  retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
  (SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
  (notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes

ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
 2026-07-23 11:45:12 -04:00
Joseph Doherty a54602c14a docs(grpc): Phase 2 live gate — PASS; site→central S&F cutover soak 
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.

Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
  logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
  bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
  (oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
  active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
  whole dead window, 216==216 distinct — zero loss, zero dupes

Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
 2026-07-23 11:31:09 -04:00
Joseph Doherty 2fa5e93c73 docs(plans): tick 1B DoD (proportionate rig gate PASS) 2026-07-22 22:38:38 -04:00
Joseph Doherty 01693b13db docs(grpc): Phase 1B live gate — PASS (proportionate); central→site rides authenticated gRPC for 3 sites; records the central-wide SiteTransport finding 2026-07-22 22:38:21 -04:00
Joseph Doherty 86ad4d5c8e feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka) 
Extract the central→site send path in CentralCommunicationActor behind a new
ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport
(Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's
27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's
subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope.

- AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim
  (the _siteClients lookup + ClusterClient.Send with the reply-to sender
  preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path).
- GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via
  SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through
  ControlPlaneCredentials, per-command deadlines set EQUAL to today's
  CommunicationService Ask timeouts (per-command, not per-group: DeploymentState
  query and TriggerSiteFailover use QueryTimeout; the two parked relays map to
  QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds;
  WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout).
- SitePairChannelProvider: per-site A/B channel pair with sticky failover
  (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover
  may have run), background failback probe to the preferred node with 1s→60s
  doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh
  loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll.

Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site
lifecycle across refreshes), ResolveDeadline pinned to each command's current
Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process
TestServers (PSK+header+deadline attached, Unavailable failover + stickiness,
failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched
(no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests
607 green with Akka default.
 2026-07-22 20:09:43 -04:00
Joseph Doherty 518c699b90 feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2) 
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.

Decisions worth recording:

- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
  Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
  handlers with the exact same "handler not available" replies; the parked
  handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
  proxy — pinned by a dispatcher test that asserts the target is the parked probe
  and NOT the dm proxy.

- Sender preservation intact. The actor's command handlers became thin
  DispatchCommand delegations that still Forward (central Ask → reply routes
  straight back); the existing SiteCommunicationActorTests pass unchanged, which
  is the regression guard for that plumbing. UnsubscribeDebugView keeps its
  fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
  synthetic UnsubscribeDebugViewAck so a unary RPC still answers.

- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
  standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
  CommitLeave; the gRPC service returns the ack, then schedules the real
  Cluster.Leave — so a caller reaching the very node about to leave still gets its
  ack instead of a broken stream. The actor path keeps today's coupled
  resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
  immaterial). Proven at both levels: a dispatcher test asserts the ack is built
  before CommitLeave runs, and a TestServer test asserts the recorded seam order
  is resolve-then-leave.

- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
  DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
  one-public-ctor invariant and its test stay green.

Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
 2026-07-22 20:07:58 -04:00
Joseph Doherty 59b13d317b feat(grpc): T1B.1 — site_command.proto + SiteCommandDtoMapper + round-trip goldens 
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.

Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).

Contract decisions worth knowing:

- Nullable COLLECTIONS ride in per-collection wrapper messages
  (DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
  RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
  and that distinction is live at the site — the same silent-data-loss class the
  transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
  and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
  AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
  TargetValueEncoded, where "wait for the empty string" is a real target, so it
  carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
  no stock wrapper). Enum translation is an explicit switch in both directions —
  never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
  enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
  rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
  return values, attribute values, tag read/write values) as a type-tagged union
  so a boxed value keeps its runtime CLR type, as it does today under Akka's
  type-preserving JSON serializer. Dates ride as invariant round-trip strings,
  not Timestamp, which would silently normalise away DateTime.Kind and
  DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
  falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
  than reusing sitestream's AlarmStateUpdate, which flattens values to display
  strings — right for a live stream, lossy for a snapshot the UI treats as
  authoritative. The encoder omits an AlarmStateChanged.Condition that already
  equals the record's derived default, so computed alarms round-trip exactly
  (record equality compares the nullable backing field, not the property).

Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).

Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
 2026-07-22 20:04:23 -04:00
Joseph Doherty aa60f43866 Merge Phase 1A: site→central control plane over gRPC (PR #26) 2026-07-22 20:01:42 -04:00
Joseph Doherty f7c7811940 docs(grpc): Phase 1A live gate — PASS (3 RPCs + restart-reconcile + coexistence); records the Kestrel-drop defect 2026-07-22 19:55:09 -04:00
Joseph Doherty 0e162cb250 fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression) 
Caught on the Phase 1A rig proof: central-a logged only "Now listening on:
http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and
the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no
startup error — the node booted, joined the cluster and served gRPC fine.

Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into
explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls
entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole
HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod),
so binding only the gRPC port silently deleted it. The site branch has the same shape
but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly.

Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2)
alongside the gRPC port (Http2) in the one ConfigureKestrel call. New
Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/
hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped).

Why no test caught it originally: unit/E2E tests use TestServer, which never binds
real Kestrel. Only a live node exposes the missing listener — which is exactly what
the rig proof is for.
 2026-07-22 19:50:32 -04:00
Joseph Doherty aa49a1d078 docs(plans): tick T1B.3/T1B.4 — central site-command transport seam complete (3be85f19) 2026-07-22 19:43:58 -04:00
Joseph Doherty 81ced76654 docs(plans): tick T1A.3/T1A.4 — site ICentralTransport seam complete (33b15f10) 2026-07-22 19:31:05 -04:00
Joseph Doherty 33b15f10a4 feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3) 
Introduce ICentralTransport as the site->central choke point inside
SiteCommunicationActor. The seven site->central sends (notification submit/
status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now
delegate to an injected transport instead of owning ClusterClient.Send inline.

- AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path,
  including the exact sender-forwarding that routes central's reply straight
  back to the waiting Ask. Default when no transport is injected -> behaviour
  unchanged, existing SiteCommunicationActorTests pass as-is.
- GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with
  sticky failover + background failback (1s-doubling-cap-60s), PSK +
  x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring
  today's Ask timeouts. Cross-node retry ONLY on provably-unsent
  connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget.
- StaticSitePskProvider: site's single own-key provider (fail-closed).
- CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints
  (validator: required when transport=Grpc). Host selects the impl; the
  ClusterClient is created only on the Akka path.

Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat
no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over
in-process TestServer (failover flip, sticky, failback, PSK+header, deadline,
no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391
green; the three above-seam suites pass unmodified.
 2026-07-22 19:29:14 -04:00
Joseph Doherty 9f2c96f486 docs(plans): tick T1B.2 — SiteCommandDispatcher + site gRPC command service (cd6c20e1) 2026-07-22 19:12:45 -04:00
Joseph Doherty fc398b47d3 docs(plans): tick T1A.2 — central CentralControlService hosting + per-site auth (780bb9c3) 2026-07-22 18:55:28 -04:00
Joseph Doherty 780bb9c369 feat(grpc): host CentralControlService on the central node (T1A.2) 
Central now ALSO listens for the seven site→central control messages over
gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet —
sites keep CentralTransport=Akka (T1A.3's job); central simply starts also
accepting.

- CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the
  SAME in-process message the ClusterClient path carries, Asks the existing
  CentralCommunicationActor (zero handler-logic changes), encodes the reply via
  the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady —
  Unavailable until AkkaHostedService hands the actor over. Heartbeat stays
  fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the
  shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status
  mapping is retry-aware: Unavailable (never dispatched, safe to cross-node
  retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere).

- CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a
  variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site
  (verify the Bearer token against the key for the site in the required
  x-scadabridge-site header, via ISitePskProvider) where a site verifies its one
  own-key — a genuinely different model. Fail-closed on every branch: missing or
  blank header, unresolvable key, and mismatched token all → PermissionDenied,
  never pass-through. One public constructor only (the explicit-prefix ctor is
  internal), pinned by a reflection test — a second public ctor makes
  Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate.

- Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort
  (default 8083, symmetric with sites), mirroring the Site branch. Additive to
  central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through
  Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with
  MapGrpcService. Port range-validated by NodeOptionsValidator.

- Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes
  so a later task can exercise it.

Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor +
real service over a stub actor) proves auth positives/negatives are
distinguishable and covers unary + the ingest bridge shapes; the interceptor is
registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the
per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests
(Communication.Tests, TestKit) covers the readiness gate, fire-and-forget
heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active
<Protobuf> item. Communication.Tests (356) + Host.Tests (384) green.
 2026-07-22 18:53:59 -04:00
Joseph Doherty c90b353820 docs(plans): tick T1B.1 — site_command.proto + mapper + 194 goldens (a7481174) 2026-07-22 18:41:22 -04:00
Joseph Doherty c615ba5f78 docs(plans): tick T1A.1 — central_control.proto + mapper + 32 goldens (d7455577) 2026-07-22 18:31:18 -04:00
Joseph Doherty d7455577a8 feat(grpc): central_control.proto + DTO mapper for the 7 site→central control RPCs (T1A.1) 
Phase 1A of the ClusterClient→gRPC migration needs a wire contract for the
seven messages SiteCommunicationActor forwards to /user/central-communication.
This lands the contract and its mapper only — hosting (T1A.2) and the site-side
transport seam (T1A.3) follow.

`Protos/central_control.proto` (package scadabridge.centralcontrol.v1, service
CentralControlService) declares SubmitNotification, QueryNotificationStatus,
IngestAuditEvents, IngestCachedTelemetry, ReconcileSite, ReportSiteHealth and
Heartbeat. Note the direction is the inverse of SiteStreamService: here the site
dials and central serves.

Decisions worth recording:

- The two ingest RPCs IMPORT sitestream.proto and reuse AuditEventBatch /
  CachedTelemetryBatch / IngestAck rather than redeclaring them. The site
  telemetry actor already builds those messages, so a second copy would fork one
  wire contract into two kept in lockstep by hand. ForwardState / IngestedAtUtc
  stay off-wire exactly as they are today.
- Heartbeat replies google.protobuf.Empty — it is fire-and-forget and must never
  surface a fault onto the heartbeat timer path.
- The three NULLABLE SiteHealthReport collections travel inside single-field
  wrapper messages (ConnectionEndpointMapDto / TagQualityMapDto /
  NodeStatusListDto). proto3 cannot express presence on repeated/map fields, but
  null and empty genuinely differ here — SiteHealthCollector emits
  `ClusterNodes: _clusterNodes?.ToList()` and the central health surface reads
  null as "not reported", not as "reported empty". Same reasoning drives the
  BoolValue/Int64Value/DoubleValue wrappers on LocalDbReplicationConnected,
  LocalDbOplogBacklog and the two age gauges, whose docs are explicit that null
  is not zero/false.
- ConnectionHealthEnum reserves 0 for UNSPECIFIED instead of mapping Connected
  onto it, and the decoder resolves anything unknown to ConnectionHealth.Error.
  An unrecognised connection state must not render as "healthy".
- Guid? execution ids travel as "D" strings with empty meaning null; a malformed
  non-empty value throws rather than being laundered into "no correlation".
- DateTimeOffset normalizes to a UTC instant (a protobuf Timestamp has no
  offset). Lossless in practice — every producer stamps UTC — and documented +
  asserted rather than left implicit.

SiteCallDtoMapper gains a ToDto(SiteCall) overload. Its doc comment previously
asserted such a method "would be dead code"; that held only while ClusterClient
was the sole path from IngestCachedTelemetryCommand (which carries SiteCall, not
SiteCallOperational) to central. Comment corrected alongside.

Golden tests round-trip every message through a real protobuf encode/decode —
DTO → proto → bytes → proto → DTO — with a fully-populated case and a
null/empty/minimal case for every optional member. Verified to have teeth by
mutation: dropping a scalar, collapsing an empty nullable collection to absent,
and nulling a gauge each fail a test.

Codegen stays CHECKED IN under CentralControlGrpc/ (protoc segfaults in the
linux_arm64 Docker image); no active <Protobuf> item is committed.
docker/regen-proto.sh is generalized to `regen-proto.sh [sitestream|
centralcontrol|all]` — it now injects the ItemGroup rather than unwrapping a
comment, so it no longer depends on there being exactly one Protobuf line, and
it restores the csproj verbatim on every exit path.
 2026-07-22 18:28:27 -04:00
Joseph Doherty aa7c5cd138 docs(plans): tick Phase 0 DoD — PR #25 merged to main @ 3fa95555 2026-07-22 18:12:01 -04:00
Joseph Doherty 3fa955556d docs(grpc): record the Playwright result and root-cause both failures 
Phase 0's gate doc now carries the full suite picture, not just the rig checks.

Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):

- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
  user-initiated transaction while the central context has EnableRetryOnFailure,
  so SqlServerRetryingExecutionStrategy refuses the split query inside it and
  bundle import fails against real MS SQL. The unit/integration suite cannot see
  it -- the in-memory EF provider has no retrying strategy and BeginTransaction
  is a no-op there.

- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
  ^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
  has also silenced everything after the toast assertion -- including the
  secret-non-leak check on the Auth Token.

Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.

Neither is fixed here; both are out of scope for a PSK-auth branch.
 2026-07-22 18:09:07 -04:00
Joseph Doherty 6ef8c7d70a docs(grpc): Phase 0 live gate PASS — record results, the inert-gate defect, and the trap for phases 1A/1B 
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.

Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
 2026-07-22 18:01:11 -04:00
Joseph Doherty 228ff8b428 fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert 
Caught by the Phase 0 live gate, not by the suite.

Grpc.AspNetCore registers the interceptor BY TYPE, and
InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all
given argument types have been found" when more than one public constructor is
applicable. The interceptor had two: the DI one and a prefix-set overload added
for later phases.

The failure mode is nasty. The throw happens inside the interceptor pipeline on
every call, so nothing fails at startup — the site node boots, joins, reports
healthy. Every gated call then dies with Unknown / "Exception was thrown by
handler", which reads as a handler bug rather than an auth bug. And it fails
OPEN in the sense that matters least and closed in the sense that matters most:
no call is ever authorized, but no call is ever correctly REFUSED either, so the
rig showed identical errors for a correct key, a wrong key and no key at all.
Live evidence, site-a: three PullAuditEvents calls, three identical
InvalidOperationExceptions in the node log.

Fix: the prefix-set constructor is internal (Host.Tests already has
InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding
a second public registration shape.

Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests
registered the interceptor with AddSingleton alongside AddGrpc, so DI handed
back the instance and Grpc.AspNetCore's activation path — the thing that throws
— never ran. The harness now registers exactly as Program.cs does, by type and
not in DI. Plus a direct reflection assertion that the type has exactly one
public constructor, since that is the real invariant and it is cheap to pin.
 2026-07-22 17:56:51 -04:00
Joseph Doherty 2ee84af1c0 feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration 
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.

T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.

T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.

T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.

Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.

Two decisions beyond the plan:

  * StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
    the runtime gate, but fail-closed with no boot check produces a node that
    joins, answers heartbeats and reports healthy while refusing every stream,
    audit pull and telemetry ingest — silent and total. Same reasoning as the
    existing inbound API-key pepper rule.

  * Added Communication:SitePsks as a central-side key map. The plan assumed
    central would read the store, seeded via a dev KEK; the docker rig
    deliberately boots with no master key, so store-only resolution would leave
    it unable to dial its own sites. The store stays primary — it is the only
    source that can serve a site added at runtime — with the map covering
    key-less hosts and one-off pins. Neither source falling back to
    "unauthenticated" is the invariant.

T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.

OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
 2026-07-22 17:51:09 -04:00
Joseph Doherty f1ad967083 docs(plans): ClusterClient→gRPC-only migration plan (phases 0–5 + tasklist) 
Complete executable plan: PSK-from-Secrets auth (Phase 0), central_control +
site_command proto contracts behind transport seams inside the two
communication actors (1A ∥ 1B in worktrees), per-direction cutover flags,
ClusterClient/receptionist deletion, 8-check live gate. Design doc:
scadaproj/scadabridge_clusterclient_to_grpc.md (§7 = deep-dive corrections).
 2026-07-22 17:10:33 -04:00
Joseph Doherty 654df8abc2 docs(cluster): site-pair manual failover runbook + component spec 2026-07-22 07:49:51 -04:00
Joseph Doherty c8e2f4da02 feat(cluster): site-pair manual failover relayed from the central UI (Task 10) 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
 2026-07-22 07:48:56 -04:00
Joseph Doherty d66e0d585f chore(plans): mark self-first ordering + manual failover tasks complete (live gate PASS) 2026-07-22 07:14:11 -04:00
Joseph Doherty caf14a3e03 feat(ui): admin manual-failover control on the health page 
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.

- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
  intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
  the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
  hand over, in-flight work on the active node is interrupted, and THIS PAGE
  will disconnect and reconnect against the new active node -- Traefik routes
  the UI to the node being restarted, so a working failover otherwise reads as
  a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
  reports a failover that did not happen.

7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.

Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
 2026-07-22 07:00:10 -04:00
Joseph Doherty f679d5c749 feat(cluster): manual central failover service — graceful Leave of the oldest Up member 
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
 2026-07-22 06:55:15 -04:00
Joseph Doherty eca69505bc docs(plans): record the self-form watchdog rejection and the self-first ordering that replaced it 2026-07-22 06:33:28 -04:00
Joseph Doherty 4a6341d871 feat(cluster): self-first seed ordering closes the boot-alone outage gap 
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
 2026-07-22 06:32:00 -04:00
Joseph Doherty 69b3ccfc37 docs(components): correct the three component docs against the code they describe 
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.

ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.

Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.

DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.

Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.

Docs only; no code changed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 18:45:56 -04:00
Joseph Doherty cf3bd52f93 feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety) 
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
 2026-07-21 10:53:40 -04:00
Joseph Doherty dced0d2794 fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — four new NU1903 advisories on the 10.0.7 DataProtection transitive broke fresh restores (docker image build) 
Same pattern as the SQLitePCLRaw pin: direct PackageReference at the chain's entry
project (ConfigurationDatabase). Bumping the DataProtection parent instead was tried
and rejected — 10.0.10 floors Microsoft.Extensions.*/EF at 10.0.10 (NU1605 cascade).
 2026-07-21 10:53:26 -04:00
Joseph Doherty 34227991ea docs(claude): correct five understatements about the inter-cluster boundary 
Found while OtOpcUa researched this repo as the model for its own per-cluster
mesh work. Each was verified against the code, not inferred from the doc.

Three transports cross the boundary, not two: ClusterClient, gRPC, and plain
token-gated HTTP for the deploy config fetch (DeploymentConfigEndpoints,
X-Deployment-Token, AllowAnonymous with the per-deployment token as the entire
security boundary).

The gRPC direction is inverted from the data flow. Data moves site to central,
but each SITE hosts the gRPC server and central dials in — MapGrpcService appears
only in the Site branch of Program.cs. There is no gRPC server on central, which
is why the two Ingest* RPCs are dead in practice. Also records the 6-RPC surface
(the doc implied 2), the (site, endpoint) factory key that fixed an arch-review
High, and the vendored-generated-code caveat.

Active/standby is ActiveNodeEvaluator.SelfIsOldestUp — the OLDEST Up member, and
explicitly never cluster.State.Leader, because leadership diverges from singleton
placement permanently after a restart-and-rejoin and both sides claim it during a
partition. The equivalence oldest-Up == singleton placement is the design, and it
was not stated anywhere in this file.

All clusters share one ActorSystem name ("scadabridge", hardcoded at
AkkaHostedService.cs:191); they are separate clusters only by seed partitioning.
Required, not incidental — Akka.Remote address matching means ClusterClient could
not reach a differently-named system. Site nodes also carry two roles, base plus
site-{SiteId}, with singletons scoped to the site-specific one.

Neither inter-cluster transport is authenticated or encrypted: no Akka TLS or
secure cookie, gRPC is h2c, and LocalDbSyncAuthInterceptor gates only the LocalDb
sync path — so the whole SiteStreamService surface, including the two Pull RPCs
returning audit rows, is reachable by anyone who can hit :8083. The boundary
assumes a trusted network; that assumption deserves to be explicit.

Also promotes two things from docs/ into CLAUDE.md because they change decisions:
the default 128 KB Akka frame size with log-frame-size-exceeding off and no custom
serializer (a silent single-message drop that leaves the association healthy), and
the registered two-node keep-oldest total-outage gap, whose own drill has a mode
existing "to make the registered gap observable — not to pretend it is covered."

Committed to a branch rather than main; merge at your discretion.
 2026-07-21 10:00:01 -04:00
Joseph Doherty 9d925c3347 Merge chore/localdb-0.1.3: pin ZB.MOM.WW.LocalDb 0.1.3 
Picks up two rebuilt-peer replication fixes found by the OtOpcUa LocalDb
Phase 1 live gate. They matter more here than in OtOpcUa, which replicates 2
tables to this repo's 10:

- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
  detection read that as 'no gap possible', so a node whose database was lost
  rejoined empty and stayed empty until the next deploy.
- 0.1.3 — with back-fill working, the rebuilt node's own writes were silently
  dropped until its restarted seq counter climbed past the peer's stale
  watermark.

Build clean; SiteRuntime 512 green.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:19:13 -04:00
Joseph Doherty 8c6fc2f886 chore(localdb): pin ZB.MOM.WW.LocalDb 0.1.3 (rebuilt-peer replication fixes) 
Two fixes, both about a node whose LocalDb file is lost. They matter more here
than in OtOpcUa, which replicates 2 tables to this repo's 10.

0.1.2 — a converged pair prunes every oplog row on ack, and snapshot detection
read an empty oplog as "no gap possible". The steady state of a healthy pair
was the one state from which a rebuilt node could never be healed: it rejoined
empty and stayed empty until the next deploy.

0.1.3 — with back-fill working, the rebuilt node's OWN writes turned out to be
silently dropped: last_applied_remote_seq is a watermark in the peer's seq
space, and a rebuilt peer numbers from 1, so the healthy node's stale watermark
made the sender skip its whole oplog.

Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig; the
second only became reachable once the first was fixed.

Build clean; SiteRuntime 512, SiteEventLogging 70 green (Host 330,
HealthMonitoring 97 green on 0.1.2, unchanged by the pin).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:18:02 -04:00
Joseph Doherty 62cddcfa56 fix(audit): let an unresolvable cached row leave the drain queue 
A cached-telemetry audit row whose OperationTracking snapshot could not be
resolved was skipped and left Pending. Nothing ever removed it, so the next
drain re-read it, failed identically, and logged again — forever.

The severity was worse than the original write-up said. The queue is read
oldest-first with a fixed BatchSize (default 256), so once a batch's worth of
permanently-unresolvable rows collected at the head, every drain re-read
exactly those rows and NEVER REACHED the newer rows behind them. That is a
permanent stall of the cached-telemetry path, not the "log flood, no data loss"
the issue was first filed as. Measured on the rig at ~2,800 warnings/minute,
surviving both a process restart and a container restart.

Fix: after a grace period (CachedTrackingGraceSeconds, default 300 s) the
operational half is abandoned and the row marked Forwarded. A row with no
CorrelationId is abandoned immediately — it can never resolve.

Marking Forwarded does not drop audit data. That state means "no longer owed by
the drain, still eligible for reconciliation", which is exactly this situation:
ReadPendingSinceAsync covers Forwarded as well as Pending and central dedups on
EventId, so the reconciliation pull still delivers the audit half. What is lost
is the operational (SiteCalls) half, which is unrecoverable anyway once the
tracking row is gone.

Three deliberate boundaries:
- Inside the grace window the row is still retried — a missing snapshot is
  normally a brief write race, and abandoning on the first failed lookup would
  discard the operational half of every cached call that lost that race.
- A tracking-store THROW never abandons, however old the row: a throw is a
  store fault (locked, corrupt, mid-restore), not a verdict about the row.
- Logging is per drain pass, not per row. Deferred rows dropped to Debug —
  being inside the grace window is normal operation, not a warning.

CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was RETARGETED
rather than deleted: it pinned the defect ("skipped and stays Pending"), so its
orphan assertion is inverted and the half that still holds is kept verbatim.
Three tests added, including the starvation regression. Both "must NOT abandon"
tests carry a positive control on the read count, so they cannot pass by virtue
of a drain that never ran.

Verified non-vacuous: with abandonment disabled the two abandonment tests go
red and the two guards stay green. Build 0 warnings; AuditLog 358, Host 330,
SiteRuntime 512, StoreAndForward 130, Communication 312, Commons 684,
Integration 94 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 06:29:22 -04:00
dohertj2 28ca04d7de LocalDb adoption Phase 1 + 2: consolidate the site database, delete the bespoke replicators (#23) 
10-check live gate PASS; 3,509 tests green; replication remains default-OFF.
 2026-07-20 06:06:04 -04:00
Joseph Doherty 7621b48925 docs(known-issues): cached-telemetry drain hot-loops on a missing tracking snapshot 
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.

The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.

Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.

No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:59:35 -04:00
Joseph Doherty 7b5a5a6f34 docs(localdb): phase 2 truth pass across both repos 
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.

Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.

Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.

Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.

Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.

Deletes the phase2 resume-state scratch doc, which said to delete it on landing.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:37:48 -04:00
Joseph Doherty 158e79bb50 docs(localdb): phase 2 live gate evidence — all 10 checks pass 
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.

Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
  __localdb_row_version HLC and origin node id, and the standby logged zero
  config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
  oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
  nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
  the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
  errors.

Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.

Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:20:54 -04:00
Joseph Doherty 3c87b11bcf docs(localdb): phase 2 live gate evidence (INCOMPLETE — 5 of 10) 
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.

Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.

Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.

Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
  WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
  the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
  aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
  Re-scraped via a network-sharing sidecar: the metrics are present.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:10:43 -04:00
Joseph Doherty 166f07fa68 chore(localdb): adopt LocalDb 0.1.1 and drop the directory shim 
LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the
SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along
with its call site. The gap was found here but was never ScadaBridge's alone —
every LocalDb consumer had it — so the fix moved to the library.

SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with
the shim. It was already written against the site registration path, not the
mechanism, so it needed only its Ensure() call removed: what a site node
requires is that resolving ILocalDb not fail on a fresh machine, regardless of
who provides that. Verified it still earns its place — pinned back to LocalDb
0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it
genuinely depends on the library behaviour and not on a coincidence.

Also corrects the coverage-split note in StoreAndForwardStorageTests, which
asserted that directory creation is "NOT LocalDb's" — true when written, wrong
as of 0.1.1.

Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:58:20 -04:00
Joseph Doherty 9ec7966dac docs(localdb): task state for tasks 17-19 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:42:16 -04:00
Joseph Doherty 921edab454 chore(docker): size the site-a oplog caps from the phase 2 soak 
MaxBatchSize 500 -> 16. The default is a ROW count, not a byte budget, so the
batch size in bytes is set by the widest replicated column — config_json, which
Task 1 measured at up to ~60-70 KB in production. 70 KB x 500 is ~35 MB against
gRPC's 4 MB default receive limit; 16 keeps a worst-case batch near 1.1 MB.

MaxOplogRows 1,000,000 -> 250,000 and MaxOplogAge 7d -> 2d, sized from the
soak's 0.80 sf_messages rows/sec (~69k rows/day). Tighter than default is
correct here because exceeding a cap is not data loss: the oplog prunes to the
ceiling and sets needs_snapshot, so the peer catches up by snapshot resync
instead of incrementally. That trades a rare full resync for a bounded file.

site-b and site-c stay unreplicated, so the default-OFF posture is still proven
side by side on one rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:47 -04:00
Joseph Doherty 15013156bf test(localdb): two-node convergence for config tables + sf_messages 
Four scenarios over the real site pair, driven through the REAL
SiteStorageService rather than hand-written SQL. The sibling suites had to
hand-write SQL because they were specifications written BEFORE the cutover;
this suite runs after it, so it can drive the shipped writers — which is what
makes the cascade scenario meaningful.

- DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts
  config_json only, so a capture that dropped or defaulted any other column
  would still pass. deployed_at matters most — SiteReconciliationActor's
  guarded write compares it.
- RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged
  highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one
  transaction, the schema has no foreign keys, and CDC ships three independent
  per-table streams. A dropped delete leaves a permanently stale override or
  alarm row on the standby, invisible until the instance name is redeployed.
  Carries a never-removed control instance, without which "the cascade
  converged" is indistinguishable from "node B lost these tables entirely".
- ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass
  if entries replicated but were never acked, and an oplog that only grows
  trips the caps into a snapshot resync.
- RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives
  property is per-table, and an unregistered table is silently local-only
  rather than an error, so it is re-proved on the tables the N1 scenario does
  not touch.

Non-vacuity verified as the plan requires: with the eight Phase 2
RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red
(4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:01 -04:00
Joseph Doherty 605e56829e chore(config): retire ReplicationEnabled, make legacy db paths migration-only 
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed
meaning or died outright:

- ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the
  property, its 10 config entries, and the 5 test references.
- SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files
  SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory
  rules are relaxed accordingly (StartupValidator's Site-only Require, and the
  S&F validator's non-empty rule) — an absent value now means "nothing to
  migrate", so an already-migrated node can drop the key. DatabaseOptions-
  Validator still rejects a present-but-blank value.
- SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor.
  Deleted with its validator rule.

The two path keys stay present in every config, now with a comment explaining
why: removing them would strand un-migrated data on a node that has not yet
started once.

Both relaxations are pinned by the inverse of the test they replace
(Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each
verified to fail with the old rule restored.

Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored
deploy/ tree, so its edit is local-only and must be repeated on the box.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:35:11 -04:00
Joseph Doherty 3364145d63 docs(localdb): resume state through the cutover (tasks 1-16 done, task 17 next) 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:22:30 -04:00
Joseph Doherty 0ad11d6b55 docs(localdb): task state for the 14/15/16 cutover 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:17 -04:00
Joseph Doherty 037798b367 feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:05 -04:00
Joseph Doherty df0c6031ba docs(localdb): task state for tasks 10-13 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:48 -04:00
Joseph Doherty 79ce51612e test(site): pin the active-node notification-config purge; scope the guarded write 
Task 12 + Task 13. No production behaviour change in either.

Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges
notification_lists and smtp_configurations on every artifact apply, but nothing
pinned the actor's CALL to it — ArtifactStorageTests covers the storage method
only. Task 15 deletes SiteReplicationActor's copy, making this the sole
remaining call site, and Task 16 edits this actor's wiring; dropping the call
would leave plaintext SMTP passwords on disk with every suite still green.
Verified red-first by commenting the call out: the pin fails with that message.

Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers —
SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166
(survives). Reconciliation is a per-node startup self-heal against central
whose fetch races real deploys, so the deployed_at guard still does real work
there. Doc comment rewritten to say so, and to warn against porting the guard
onto the replication path where it would fight the HLC rather than help it.
Also corrected a stale 'guarded standby write' section header in the tests.

Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production
reader remains SiteReplicationActor:157, so its option + validator rule stay
until Task 17, after Task 15 deletes the actor.

Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153,
LocalDb integration 16 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:00 -04:00
Joseph Doherty c56bf4ae65 test(localdb): port resync + config replication intents as CDC convergence specs 
Task 11. Companion to Task 10, covering the intents held by
SiteReplicationActorTests and SfBufferResyncPredicateTests.

N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate
existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a
wrong-direction resync wiped a live buffer and the code needed an oldest-Up
predicate to decide authority. LocalDb's snapshot resync merges per row under
LWW and never deletes, so this asserts the property the guard protected — no
node loses rows to a peer's snapshot — with no directional-authority assertion,
because there is no active/standby asymmetry left to enforce. The setup is the
wipe scenario made concrete: each node writes rows the other never sees while
partitioned, plus a contended row, then they resync.

DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double
recording zero invocations; this harness has no actor system, no central and no
IDeploymentConfigFetcher in the graph, so the double would record zero calls
whether or not the fetch path still existed. An assertion that cannot fail is
worse than none, so the test proves the positive half — config reaches B over
replication alone — and the file records that the negative half is proved by
Task 15 deleting the code and the build still passing. It also records the D1
scope note: SiteReconciliationActor survives and legitimately fetches at
startup, so 'the standby never fetches, ever' would be false.

Non-vacuity verified by unregistering deployed_configurations: all 4 fail.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:54:42 -04:00
Joseph Doherty 2bbe66311d test(localdb): port store-and-forward replication intents as CDC convergence specs 
Task 10. Specifications first, deletion second: the bespoke ReplicationService
(explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour
is restated here as outcomes the CDC replacement must still deliver. Written
in terms of ROWS, not operations — under CDC there is no Add or Park message to
observe, only a row that must end up right on both nodes.

Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the
mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous
and batched by construction. Its portable content is the ordering OUTCOME —
add-then-remove must never converge to present — which is a test here, with
that reasoning recorded in the file so it does not read as an accidental drop.

DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather
than duplicating ~150 lines. Phase 1's tests now derive from it and still pass
unchanged. The harness registers the Phase 2 tables itself, since production
OnReady does not until Task 14; that method is marked for deletion at the
cutover, and the 8-table list is written literally so a cutover registering the
wrong set fails these tests instead of agreeing with itself.

Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th —
the ordering test — PASSED, because an absent row is also what a pair that
replicates nothing looks like. Fixed with a control row that must converge in
the same window, so the absence is evidence rather than silence.

Also corrected two comments from Task 9 that claimed Task 14 makes
notification_lists/smtp_configurations replicated. It explicitly does not
register them, for the same reason the migrator skips them.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:50:48 -04:00
Joseph Doherty 0dbfefba62 docs(localdb): task state for tasks 7-9 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:59 -04:00
Joseph Doherty 5ddc7eed6d feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB 
Task 9. Seven of the nine site configuration tables are copied out of the
legacy scadabridge.db in a single transaction, with one rename at the end: a
partial config migration would leave a site node running against half its old
configuration, which is worse than failing startup outright.

notification_lists and smtp_configurations are deliberately NOT migrated.
Both are purged on every deploy and permanently empty by design since the
site write paths were removed (2026-07-10), but a pre-fix legacy file can
still hold rows — and smtp_configurations.password is plaintext. Task 14
makes these tables replicated, so migrating them would push plaintext SMTP
passwords across a channel whose only historical payload was exactly that.
The tables are still created; only their historical contents stay behind.

Generalizes Task 8's MigrateTable into MigrateFile(many tables, one
transaction, one rename), keeping the legacy/current column intersection.

Five tests, including a column-parity test asserting each declared column
list equals the live SiteStorageSchema's. That mismatch is otherwise
invisible: a typo'd column is silently dropped by the intersection, and a
missing one silently leaves data behind. Non-vacuity verified twice — once by
removing the Migrate call (3 fail), once by wrongly adding notification_lists
and smtp_configurations to the table map (the skip test fails).

Verified: solution build 0 warnings; Host 329, SiteRuntime 532,
StoreAndForward 153 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:31 -04:00
Joseph Doherty bdc0dffea2 feat(localdb): migrate legacy store-and-forward.db into the consolidated DB 
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a
third call in Migrate alongside the two Phase 1 files.

Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data
volume, so a real deployment has a real file here holding undelivered
messages. This migration genuinely moves data; losing it would discard exactly
the buffered calls store-and-forward exists to protect.

No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key,
so INSERT OR IGNORE is idempotent across a crash-then-rerun.

The copy intersects the legacy column set with the current one rather than
naming all 16 columns outright. A file from an older build predates
execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing
column throws 'no such column' — which the existing reader treats as an
unrecognised shape and silently discards every row. A required-column (PK)
guard keeps that tolerance from degrading into copying NULL-keyed rows.

Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog
assertion that catches migrate-before-register, and the older-column-set case.
All four verified to fail without the Migrate call.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:32:10 -04:00
Joseph Doherty f8aa02e2a9 feat(localdb): create config + sf_messages tables in the consolidated DB 
Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and
StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site
configuration tables and the store-and-forward buffer live in the
consolidated LocalDb file.

Deliberately NOT registered for replication. The bespoke SiteReplicationActor
and the StoreAndForward ReplicationService still own these tables until the
Task 14 cutover deletes both and registers them in one commit; registering
early would run two replicators over the same rows and let either one's
defects hide behind the other's writes.

Pinned by two tests through the real composition root: one asserting all
twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1
pair. Non-vacuity verified by removing the DDL and observing the failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:28:05 -04:00
Joseph Doherty fefbbb31da docs(localdb): resume state through wave 2 (tasks 1-6 done, wave 3 next) 
Records the latent Phase 1 directory defect and its open library-vs-app decision,
the CreateConnection contract change, the new TestSupport library, and the
verification numbers at the wave-2 boundary.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:14:18 -04:00
Joseph Doherty 19ab0ac913 chore(localdb): record tasks 5-6 completion, the latent directory defect, and the CreateConnection contract change 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:06:06 -04:00
Joseph Doherty f2efeb37b7 refactor(sf,site): both stores take ILocalDb instead of a connection string 
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.

StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.

DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.

Two things the plan did not anticipate, both worth reading:

1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
   directory creation simply moved to LocalDb along with file ownership. It did
   not: the LocalDb library never creates the parent directory, and
   SqliteLocalDb opens the file eagerly in its constructor — so a missing
   directory is a hard boot failure ("SQLite Error 14: unable to open database
   file"), not a degraded start. The default site config points at the RELATIVE
   path ./data/site-localdb.db, so any site node without a pre-existing data/
   directory fails to boot. The docker rig escapes only because its volume mount
   happens to create /app/data — a coincidence that would have hidden this until
   a bare-metal or fresh deployment. This has been latent since Phase 1 made
   LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
   would have widened it. Re-established the guarantee at the layer that now
   owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
   pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
   tests written against the wrong assumption failed with exactly this
   SQLite Error 14 before the fix existed.

2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
   project; the constructor change actually reaches 40 files across 7 test
   projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
   equivalent for, so every one had to move to a real temp file. Rather than
   copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
   tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
   WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.

Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.

Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:05:45 -04:00
Joseph Doherty 3dfb288b74 chore(localdb): record tasks 3-4 completion and their deviations in the task state 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:21:09 -04:00
Joseph Doherty ac5eb12cce refactor(site): extract SiteStorageSchema.Apply from SiteStorageService 
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.

Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.

One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.

Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.

SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:20:50 -04:00
Joseph Doherty 9e3239c5d9 refactor(sf): extract StoreAndForwardSchema.Apply from the storage class 
Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages
DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index
move verbatim into a static sync Apply(SqliteConnection), depending only on
Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers; the store
still calls it so a directly-constructed store stays self-sufficient.

Two deviations from the plan as written, both deliberate:

1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4
   snippet drops it, but Task 4 explicitly says not to move the equivalent
   pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict
   each other. Keeping it preserves behaviour for the intermediate commits and
   for directly-constructed stores; it becomes moot in Task 5 when the store
   stops opening its own connections. Dropping it now would quietly regress the
   concurrent-writer support the pragma's own comment documents.

2. Added a second test beyond the plan's. The specified test asserts only
   against a freshly-created table, where CREATE TABLE already lists all 16
   columns — it would pass with every ALTER deleted. The added test starts from
   the pre-upgrade 12-column shape with a row in it and proves the upgrade path
   runs and preserves data.

That second test also surfaced that the backfill lands 1 ms low: julianday()'s
double day-fraction cannot represent every millisecond. Pre-existing behaviour,
carried over verbatim, asserted with a 1 ms tolerance rather than pinning a
precision the implementation never had.

Suite: 154 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:18:05 -04:00
Joseph Doherty 12eb97e07a docs(localdb): close the phase 2 gate — questions answered, plan written 
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:15:53 -04:00
Joseph Doherty d512572dac docs(localdb): phase 2 resume state — where the plan stands and what carries forward 
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:42 -04:00
Joseph Doherty 8652eab98e fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups 
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.

Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.

The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.

Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
  LocalDb-adjacent catch sites (the missing extended code is what made the
  original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
  (NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
  mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
  volume hung the reseed forever; compose mounts annotated as informational.

Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.

Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:00 -04:00
Joseph Doherty e9e11d635e docs(known-issues): root-cause brief for the LocalDb disk-I/O-error-under-load defect 
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.

Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:11:08 -04:00
Joseph Doherty 82c869a1df docs(localdb): record the phase 2 task-1 gate verdict in the task record 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:07:18 -04:00
Joseph Doherty fc553bd9ba docs(localdb): phase 2 rig soak findings — GATE FAILS on a phase 1 defect 
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.

BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).

Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.

Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:06:59 -04:00
Joseph Doherty cf46e59680 fix(rig): seed LDAP group mappings with the canonical role names 
docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role
strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'.
The mismatch authorized nothing: on a freshly reseeded rig every
Designer/Deployer-gated management command failed UNAUTHORIZED, including
seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d
encrypted-secret restore.

Found while standing up the Phase 2 soak rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:55:04 -04:00
Joseph Doherty 25463d522f docs(localdb): phase 2 plan review pass — corrections from code verification 
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:27:11 -04:00
Joseph Doherty f6ca82a9e2 docs(localdb): phase 2 implementation plan (21 tasks, full scope) 
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.

Resolves the phase 2 gate's five open questions from code recon (D1-D5):

D1 notify-and-fetch is DELETED, not preserved. It exists only because the
   config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
   deployed_at version guard protected against a stale fetch racing, so it
   dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
   unnecessary: LocalDb's snapshot resync merges per-row LWW and never
   deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
   lower-HLC incoming row). Semantic change - the standby is convergent,
   no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
   re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
   gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
   must stop and start together.

Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 15:04:37 -04:00
565 changed files with 84396 additions and 13601 deletions
Expand all files Collapse all files
.gitignore
+1
View File
@@ -55,3 +55,4 @@ docker-env2/*/data/
# Sister-project deployment artifacts (not part of this solution)
/deploy/
email_details.txt
CLAUDE.md
+37 -69
View File
@@ -8,29 +8,12 @@ When a change is requested, the default assumption is: update the design doc *an
### Top-level directories
- `src/` — C#/.NET implementation, one project per component (e.g. `ZB.MOM.WW.ScadaBridge.AuditLog`, `ZB.MOM.WW.ScadaBridge.NotificationOutbox`, `ZB.MOM.WW.ScadaBridge.SiteCallAudit`, `ZB.MOM.WW.ScadaBridge.CentralUI`, `ZB.MOM.WW.ScadaBridge.Host`, …). Solution file: `ZB.MOM.WW.ScadaBridge.slnx`.
- `tests/` — Test projects (unit + integration).
- `docs/` — Design documentation: `docs/requirements/` (high-level + per-component specs), `docs/test_infra/` (test infrastructure), `docs/plans/` (design-decision and implementation-plan docs). The spec the code implements.
- `docker/` — 8-node cluster topology (2 central + 3 sites), `deploy.sh`, per-node `appsettings.*.json`. See [`docker/README.md`](docker/README.md) for setup, ports, and management commands. Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — Minimal second cluster topology (2 central + 1 site × 2 nodes), runs concurrently with `docker/` on host ports 91XX. Built specifically for testing the Transport (#24) feature with two real environments. See [`docker-env2/README.md`](docker-env2/README.md). Rebuild + redeploy with `bash docker-env2/deploy.sh`.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`).
- `deploy/` — Production/on-host deployment artifacts (e.g. `deploy/wonder-app-vd03/`: `appsettings.Central.json`, `appsettings.Site.json`, `install.ps1`/`uninstall.ps1`, `RUNBOOK.md`).
- `deployments/` — Deployment topology notes (`docker-cluster.md`, `docker-cluster-env2.md`, `README.md`).
- `code-reviews/` — Per-component code-review notes (one folder per component, plus `_template`).
- `tools/` — Repo maintenance/utility scripts (e.g. `rename-to-scadabridge.sh`).
- `AkkaDotNet/` — Akka.NET reference documentation and best-practices notes.
- `deprecated/` — Retired docs/notes kept for reference.
- `logs/` — Local runtime log output.
- `vendor/` — Vendored third-party assets (currently an empty placeholder).
- `.claude/` — Claude Code project config (settings, skills, agents).
Layout is self-describing (`ls`, plus each directory's own README). Only the non-obvious parts are recorded here:
### Key documents
- `README.md` — Master index with component table and architecture diagrams.
- `docs/requirements/HighLevelReqs.md` — Complete high-level requirements covering all functional areas.
- `docs/requirements/Component-*.md` — Individual component design documents (one per component) — the spec the code implements.
- `docs/test_infra/test_infra.md` — Master test infrastructure doc (OPC UA, MS SQL, SMTP, REST API, Traefik). LDAP is the shared GLAuth on `10.100.0.35:3893` (not a local infra container; see `scadaproj/infra/glauth/`).
- `docs/plans/` — Design decision and implementation-plan documents from refinement sessions.
- `docker/` — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — minimal second cluster (2 central + 1 site × 2 nodes) that runs **concurrently** with `docker/` on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with `bash docker-env2/deploy.sh`.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`), and the same applies to `docs/test_infra/test_infra.md`, which still reads as if it were a local container.
- `deprecated/` — retired docs/notes kept only for reference; not current.
## Sister Projects
@@ -70,35 +53,11 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Run tests with `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. `ZB.MOM.WW.ScadaBridge.CLI.Tests` is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
- **Propagate cross-repo changes to the umbrella index.** This repo is indexed by the parent workspace `~/Desktop/scadaproj`. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the **ScadaBridge entry in [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. `MxAccessGateway`/`HistorianGateway`.)
## Current Component List (27 components)
## Component List
1. Template Engine — Template modeling, inheritance, composition, validation, flattening, diffs.
2. Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle.
3. Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream.
4. Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
5. Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
6. Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
7. External System Gateway — External system definitions, API method invocation, database connections.
8. Notification Service — Central-only notification-list and SMTP definitions, per-type delivery adapters (sites no longer deliver notifications).
9. Central UI — Web-based management interface, all workflows.
10. Security & Auth — LDAP/AD authentication, role-based authorization, site-scoped permissions.
11. Health Monitoring — Site health metrics collection and central reporting.
12. Site Event Logging — Local operational event logs at sites with central query access.
13. Cluster Infrastructure — Akka.NET cluster setup, active/standby failover, singleton support.
14. Inbound API — Web API for external systems, API key auth, script-based implementations.
15. Host — Single deployable binary, role-based component registration, Akka.NET bootstrap.
16. Commons — Shared types, POCO entity classes, repository interfaces, message contracts.
17. Configuration Database — EF Core data access layer, repositories, unit-of-work, audit logging (IAuditService), migrations.
18. Management Service — Akka.NET actor providing programmatic access to all admin operations, ClusterClientReceptionist registration.
19. CLI — Command-line tool using HTTP Management API, System.CommandLine, JSON/table output.
20. Traefik Proxy — Reverse proxy/load balancer fronting central cluster, active node routing via `/health/active`, automatic failover.
21. Notification Outbox — Central component ingesting store-and-forwarded notifications, `Notifications` audit table, dispatcher loop, retry/parking, delivery KPIs.
22. Site Call Audit — Central component auditing site cached calls (`CachedCall`/`CachedWrite`); `SiteCalls` audit table, telemetry ingest, reconciliation, KPIs, central→site Retry/Discard relay; sites remain the source of truth.
23. Audit Log — Central append-only AuditLog table spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site SQLite hot-path + gRPC telemetry + reconciliation; combined telemetry with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API; monthly partitioning, 365-day retention.
24. Transport — File-based, encrypted bundle export/import via Central UI. Templates (with **all** child collections — attributes/alarms/scripts/compositions/**native-alarm-sources**, field-complete incl. `LockedInDerived` + script cadence/timeout), system artifacts, central-only configuration, plus site/instance-scoped config (`Site`s, site `DataConnection`s, `Instance`s + real `AreaName` by name) reconciled across environments by a `BundleNameMap` name-mapping subsystem. Per-conflict resolution with a per-line Myers diff. Runs the **script trust gate** at import review (5th `ScriptTrustValidator` call site — forbidden-API scripts rejected pre-runtime; covers template/shared/ApiMethod bodies, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** — all hard-block; template-script name-resolution findings are advisory warnings, ApiMethod findings hard-block). Correlated audit via `BundleImportId`. Never touches site runtime nodes (imported instances land `NotDeployed`).
25. Script Analysis — Shared authoritative script-trust analyzer: unified forbidden-API deny-list (`ScriptTrustPolicy`), fused semantic+syntactic validator (`ScriptTrustValidator`), Roslyn compile wrapper (`RoslynScriptCompiler`), and compile-only globals stubs (`ScriptCompileSurface`/`TriggerCompileSurface`); consumed by Template Engine, Site Runtime, Inbound API, and Central UI.
26. KPI History — Reusable central KPI-history backbone: tall/EAV `KpiSample` store in central MS SQL, `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated) sampling DI-registered `IKpiSampleSource`s every minute, bucketed query (`GetRawSeriesAsync` + `KpiSeriesBucketer`) + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart`; ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health.
27. DelmiaNotifier — Standalone external client tool (NOT a cluster component, NOT in the Host): a compact Native-AOT (`win-x64`) console app (`WWNotifier.exe`) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API `DelmiaRecipeDownload` method (`X-API-Key`, key from `SCADABRIDGE_API_KEY`), with connect-failure-only failover across a comma-list of base URLs, and reports the legacy `YES`/`NO` + exit-code stdout contract — a drop-in replacement for the legacy `WWNotifier` (see `docs/former-api-specs/dnc/`). Zero-dependency BCL-only, `System.Text.Json` source-gen. Project README: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`; design: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`.
27 components. The catalog — what each one is and its key design commitments — lives in the
`scadabridge-components` skill (`.claude/skills/scadabridge-components/SKILL.md`); the authoritative
spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carries the component table.
## Key Design Decisions (for context across sessions)
@@ -120,23 +79,34 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1, 2026-07-19).** `OperationTracking` and `site_events` now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file, configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
- **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
### External Integrations
- External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
- Dual call modes: `ExternalSystem.Call()` (synchronous) and `ExternalSystem.CachedCall()` (store-and-forward on transient failure).
- Error classification: HTTP 5xx/408/429/connection errors = transient; other 4xx = permanent (returned to script).
- Notification Service: SMTP with OAuth2 Client Credentials (Microsoft 365) or Basic Auth. BCC delivery, plain text.
- Email delivery has **two transports**, selected per config row by `SmtpConfiguration.Transport` (null/`Smtp` = default): SMTP (MailKit, Basic/OAuth2) or on-prem Exchange **EWS** (no-SDK `CreateItem` SOAP over `HttpClient`, Basic-over-HTTPS only — https enforced at write gate, adapter AND sender — BCC-only recipients, `SendOnly` so no Sent-Items copy). Under `Ews`, `Host` is the full EWS URL, `Credentials` is `username:password`, and Port/TlsMode/OAuth2*/MaxConcurrentConnections are unused. Site-facing behavior is unchanged (delivery stays central-only). Q12 (M365 OAuth2 tenant) is closed as **superseded** — the OAuth2 SMTP path stays config-selectable but untested against a live tenant. Design: `docs/plans/2026-08-10-ews-email-transport-design.md`.
- Notification delivery is central-only: sites store-and-forward notifications to the central cluster (target = central, not SMTP); sites never talk to SMTP. Notification lists and SMTP config are no longer deployed to sites; recipient resolution happens at central, at delivery time.
- Notification lists carry a `Type` discriminator (`Email` and `Sms`). `Notify.To("list")` is type-agnostic; delivery is via per-type `INotificationDeliveryAdapter` (Email via SMTP; Sms via Twilio REST — `SmsNotificationDeliveryAdapter`, no SDK, one POST per recipient, per-recipient rollup). List Type is fixed after creation.
- `Notify.Send` is async and **enqueue-only** — it buffers the notification into the local SQLite S&F store and returns a `NotificationId` (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (`deferToSweep: true` buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with `maxRetries: 0` (the "no limit" escape hatch), so notifications retry until central acks and are **never parked for retry exhaustion** — only a corrupt payload parks them (arch-review 02, Tasks 13/14). `Notify.Status(notificationId)` returns a status record (status, retry count, last error, key timestamps); answered site-locally as `Forwarding` while still in the site S&F buffer, otherwise round-trips to central.
@@ -181,7 +151,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Scope = script trust boundary: outbound API (sync + cached), outbound DB (sync + cached), notifications, inbound API. Framework/internal traffic is explicitly excluded.
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. Tag-cascade coverage is complete as of M5.4 (T4) — no further spawn points are deferred.
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost.
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
@@ -201,16 +171,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
### Cluster & Failover
- Keep-oldest split-brain resolver with `down-if-alone = on`, 15s stable-after.
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
### UI & Monitoring
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
- UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
- Dead letter monitoring as a health metric.
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
@@ -233,20 +206,15 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
### Akka.NET Conventions
- Tell for hot-path internal communication; Ask reserved for system boundaries.
- ClusterClient for cross-cluster communication; ClusterClientReceptionist for service discovery across cluster boundaries.
- Cross-cluster communication is gRPC (per-site PSK-authenticated): site→central `CentralControlService`, central→site `SiteCommandService`, plus the `SiteStreamService` data stream. ClusterClient/ClusterClientReceptionist were removed in the migration's Phase 4 — service discovery is by dialling configured endpoints, not the receptionist. (Akka.Cluster.Tools remains for ClusterSingleton.)
- Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — `ScriptTrustPolicy` is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to `ScriptTrustValidator`. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory.
- Application-level correlation IDs on all request/response messages.
## Tool Usage
- When consulting with the Codex MCP tool, use model `gpt-5.4`.
- When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.
### CLI Quick Reference (Docker / OrbStack)
- **Management URL**: `http://localhost:9000` — the CLI connects via the Traefik load balancer, which routes to the active central node. Direct access: central-a on port 9001, central-b on port 9002.
- **Test user**: `--username multi-role --password password` — has Admin, Design, and Deployment roles. The `admin` user only has the Admin role and cannot create templates, data connections, or deploy.
- **Config file**: `~/.scadabridge/config.json` — stores `managementUrl` and default format. See `docker/README.md` for a ready-to-use test config.
- **Rebuild cluster**: `bash docker/deploy.sh` — builds the `scadabridge:latest` image and recreates all containers. Run this after code changes to ManagementActor, Host, or any server-side component.
- **Infrastructure services**: `cd infra && docker compose up -d` — starts MS SQL, OPC UA, SMTP, and REST API. These are separate from the cluster containers in `docker/`. **LDAP is NOT started here** — it is the shared GLAuth on `10.100.0.35:3893` (dc=zb,dc=local); source of truth and config: `scadaproj/infra/glauth/`.
- **All test LDAP passwords**: `password` (see `scadaproj/infra/glauth/config.toml` for users and groups; canonical cross-app login: `multi-role`).
Management URL, test credentials, rebuild/redeploy commands and the infra services are in the
`scadabridge-cluster-ops` skill (`.claude/skills/scadabridge-cluster-ops/SKILL.md`).
Directory.Packages.props
+33 -11
View File
@@ -90,9 +90,9 @@
to mark tests as Skipped (not silently Passed) when MSSQL is unreachable.
-->
<PackageVersion Include="Xunit.SkippableFact" Version="1.5.61" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
@@ -103,14 +103,15 @@
<PackageVersion Include="ZB.MOM.WW.Auth.ApiKeys" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
</ItemGroup>
<!--
@@ -145,6 +146,27 @@
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
</ItemGroup>
<!--
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h,
GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against
System.Security.Cryptography.Xml 10.0.7, pulled in TRANSITIVELY by
Microsoft.AspNetCore.DataProtection 10.0.7 (ConfigurationDatabase's DataProtection
key storage). With TreatWarningsAsErrors any FRESH restore — notably the docker
image build — went red (surfaced 2026-07-21; local builds had cached audit data).
Same pattern as SQLitePCLRaw above: pin the vulnerable transitive package to its
patched version (10.0.10) with an explicit <PackageReference> in the one project
where the chain enters (ConfigurationDatabase; every other resolver — AuditLog,
SiteCallAudit, Transport, PerformanceTests, tests — reaches it through that
ProjectReference). Bumping the DataProtection parent instead was tried and
rejected: 10.0.10 floors Microsoft.Extensions.* and (via the EFCore adapter)
Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump
(NU1605 downgrade errors) that belongs in its own reviewed commit.
-->
<ItemGroup>
<PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
</ItemGroup>
<!--
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
README.md
+2 -2
View File
@@ -16,7 +16,7 @@ This repository is the full **implementation** project for ScadaBridge — the C
| Central Database | MS SQL Server, Entity Framework Core |
| Site Storage | SQLite (deployed configs, S&F buffer, event logs) |
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365; SMS via Twilio REST); store-and-forwarded from sites |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
| Hosting | Windows Server, Windows Service |
| Cluster | Akka.NET Cluster (active/standby, keep-oldest SBR) |
| Logging | Serilog (structured) |
@@ -83,7 +83,7 @@ Both stacks share the infrastructure services in [`infra/`](infra/) (MS SQL, LDA
| 5 | Central–Site Communication | [docs/requirements/Component-Communication.md](docs/requirements/Component-Communication.md) | Dual transport: Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data). 9 message patterns with per-pattern timeouts, SiteStreamGrpcServer/Client, application-level correlation IDs, transport heartbeat config, gRPC keepalive, message ordering, connection failure behavior. The gRPC stream additively carries the read-only native alarm mirror (computed + native OPC UA / MxAccess) via the enriched `AlarmStateUpdate`. |
| 6 | Store-and-Forward Engine | [docs/requirements/Component-StoreAndForward.md](docs/requirements/Component-StoreAndForward.md) | Buffering (transient failures only), fixed-interval retry, parking, async best-effort replication, SQLite persistence at sites. |
| 7 | External System Gateway | [docs/requirements/Component-ExternalSystemGateway.md](docs/requirements/Component-ExternalSystemGateway.md) | HTTP/REST + JSON, API key/Basic Auth, per-system timeout, dual call modes (Call/CachedCall), transient/permanent error classification, dedicated blocking I/O dispatcher, ADO.NET connection pooling. |
| 8 | Notification Service | [docs/requirements/Component-NotificationService.md](docs/requirements/Component-NotificationService.md) | Central-only — manages typed notification-list, SMTP, and SMS definitions; supplies per-type delivery adapters (Email via SMTP with OAuth2 (M365) or Basic Auth, BCC, plain text; SMS via Twilio REST, per-recipient, outbound-only); delivery performed by the Notification Outbox. |
| 8 | Notification Service | [docs/requirements/Component-NotificationService.md](docs/requirements/Component-NotificationService.md) | Central-only — manages typed notification-list, SMTP, and SMS definitions; supplies per-type delivery adapters (Email via SMTP with OAuth2 (M365) or Basic Auth, or on-prem Exchange EWS, BCC, plain text; SMS via Twilio REST, per-recipient, outbound-only); delivery performed by the Notification Outbox. |
| 9 | Central UI | [docs/requirements/Component-CentralUI.md](docs/requirements/Component-CentralUI.md) | Blazor Server with SignalR real-time push, load balancer failover with JWT, all management workflows. Custom-content modal host (`DialogService.ShowAsync<T>`) with focus-trap/restore; dark-mode CSS-variable token layer (`[data-bs-theme="dark"]` overriding `ZB.MOM.WW.Theme` tokens in `site.css`, `localStorage`-persisted, SSR no-flash); reusable presentational components `OffsetPager`, `KeysetPager`, and `DateTimeRangeFilter` adopted across report/audit pages. |
| 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. |
| 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
ScadaBridge-docs-fixed.md
-116
View File
@@ -1,116 +0,0 @@
# Documentation Analysis Report
Files Scanned: 793
Files With Issues: 10
Total Issues: 11
## Issues
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/BrowserTime.cs
LINE: 26
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: UTC-5
MESSAGE: Comment contains 'UTC-5', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/TransportImport.razor.cs
LINE: 225
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: Step-2
MESSAGE: Comment contains 'Step-2', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/ConnectionHealthQueryService.cs
LINE: 18
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: PLC-1
MESSAGE: Comment contains 'PLC-1', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditExportHelpers.cs
LINE: 156
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: non-403
MESSAGE: Comment contains 'non-403', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/BundleCommands.cs
LINE: 407
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: non-403
MESSAGE: Comment contains 'non-403', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs
LINE: 43
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: sub-100
MESSAGE: Comment contains 'sub-100', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs
LINE: 29
CATEGORY: TaskReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: year-0001
MESSAGE: Comment contains 'year-0001', which looks like a task/issue tracking identifier; tracking IDs should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs
LINE: 2034
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs
LINE: 206
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationService/MailKitSmtpClientWrapper.cs
LINE: 9
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
---
FILE: /Users/dohertj2/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.NotificationService/MailKitSmtpClientWrapper.cs
LINE: 113
CATEGORY: TrackingReferenceInComment
SEVERITY: Warning
MEMBER: Comment
SIGNATURE: M365
MESSAGE: Comment contains 'M365', which looks like a project tracking reference; bookkeeping references should not appear in code documentation.
ScadaBridge-docs-issues.md
-4296
View File
File diff suppressed because it is too large Load Diff
ZB.MOM.WW.ScadaBridge.slnx
+1
View File
@@ -36,6 +36,7 @@
<Project Path="tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests.csproj" />
alog.md
+9 -2
View File
@@ -108,7 +108,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `EventId` | `uniqueidentifier` PK | Generated where the event originates (site or central). Idempotency key. |
| `OccurredAtUtc` | `datetime2` | When the event happened (call returned, retry attempted, etc.). |
| `IngestedAtUtc` | `datetime2` | When central persisted the row (lags `OccurredAtUtc` for site-originated rows). |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound`. |
| `Channel` | `varchar(32)` | `ApiOutbound` \| `DbOutbound` \| `Notification` \| `ApiInbound` \| `SecuredWrite` \| `Cluster`. The last two are not script trust-boundary crossings: `SecuredWrite` records the two-person write lifecycle, and `Cluster` records operator-initiated topology actions (admin-triggered manual failover, decision 2026-07-22). |
| `Kind` | `varchar(32)` | Event kind discriminator (see kinds list below). |
| `CorrelationId` | `uniqueidentifier` NULL | Ties multi-event operations together. `TrackedOperationId` for cached calls, `NotificationId` for notifications, request-id for inbound API. NULL for sync one-shot calls. |
| `SourceSiteId` | `varchar(64)` NULL | NULL for central-originated events (inbound API, central notification dispatch). |
@@ -135,7 +135,7 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
- `IX_AuditLog_Target_Occurred (Target, OccurredAtUtc)` — "what did we send to system X."
- Partitioning by month on `OccurredAtUtc` from day one (purge becomes a partition switch instead of a delete storm).
**`Kind` values (flat — 10 discriminators across all channels):**
**`Kind` values (flat — 17 discriminators across all channels; pinned by `AuditEnumTests`):**
| Kind | Fires when |
|---|---|
@@ -149,6 +149,13 @@ Single wide table, polymorphic by `Channel` + `Kind` discriminators, JSON payloa
| `InboundAuthFailure` | An inbound API request was rejected at the auth boundary (bad/missing key). One row, `Status=Failed`, `HttpStatus=401`. |
| `CachedSubmit` | Script-side enqueue of a cached call (`ExternalSystem.CachedCall` / `Database.CachedWrite`); first row in the cached-call lifecycle, written to site SQLite before any forward attempt. |
| `CachedResolve` | Terminal row for a cached operation — `Status` = `Delivered` / `Failed` / `Parked` / `Discarded`. |
| `SecuredWriteSubmit` | An Operator submitted a two-person secured write; row written after the `PendingSecuredWrite` is persisted so it carries the store-assigned id as `CorrelationId`. |
| `SecuredWriteApprove` | A distinct Verifier approved a pending secured write (no self-approval; enforced server-side). |
| `SecuredWriteReject` | A Verifier rejected a pending secured write. |
| `SecuredWriteExecute` | An approved secured write was relayed to the site MxGateway connection. |
| `SecuredWriteExpire` | A `Pending` secured write aged past its server-side TTL and was transitioned to `Expired` without executing — emitted by the system (no verifier). |
| `ReconciliationAbandoned` | A reconciliation pull row failed to insert up to the permanent-abandon threshold and central advanced its cursor past it; one synthetic row so the loss is queryable in the Audit Log itself. |
| `ManualFailover` | An administrator triggered a manual failover of the central pair from the Health page; one row per invocation, written BEFORE the graceful `Cluster.Leave` is issued. `Target` = the leaving node's address. |
### Site: `AuditLog` (SQLite)
archreview/plans/00-MASTER-TRACKER.md
+3 -1
View File
@@ -16,7 +16,7 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|------|--------|------:|-----:|--------|-------------------|
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 10 | ✅ Merged — T2 live-drill deferred | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 11 | ✅ Merged — T2 RESOLVED 2026-08-01 (see note below) | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
| [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 15 | ✅ Merged | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 |
| [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 7 | ✅ Merged | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 |
| [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 13 | ✅ Merged | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 |
@@ -27,6 +27,8 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks
**Round-2 progress: COMPLETE — all 8 plans executed (TDD, per-task commits) and merged to `main` @ `1930f19b` on 2026-07-13 (fast-forward of the `r2-integration` assembly; origin pushed; PRs #6–#13 closed). ~81 of 86 tasks landed; the remainder are human/environment-gated: R2-08 T1 (rotate wonder-app-vd03 API key), T2/T11/T13 (delete untracked live-credential + generated files), and R2-01 T2 (live docker failover drill — in-process envelope already measured at 33.7s). Integration surfaced + fixed 2 cross-plan regressions no per-project run caught: R2-08 NodeName validator vs IntegrationTests host-boot, and R2-03 compile-cache counter under parallel test load.**
> **Update 2026-08-07 (truth sweep):** the R2-01 T2 live failover drill is **RESOLVED as of 2026-08-01** — delivered via PLAN-R2-01 T4's live `FailoverTimingTests` on the real two-node in-process rig at production timings, plus `docker/failover-drill.sh` and `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs` (`AutoDown_HardCrashOfOldestNode_*`) covering the oldest-crash direction. Recorded in the deferred-work register (`docs/plans/2026-07-08-deferred-work-register.md`, "Failover-timing measurement" row, RESOLVED 2026-08-01). R2-08 T11 (delete root `deferred.md`) and T13 (delete generated root docs reports) were completed by this sweep (the files were in fact tracked, so they were `git rm`'d). Remaining human-gated residuals: R2-08 T1 (key rotation) and T2 (delete credential files).
## Round-2 P0 (do first, any order)
1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`.
archreview/plans/PLAN-R2-01-cluster-host-failover.md.tasks.json
+13 -12
View File
@@ -4,13 +4,13 @@
{
"id": 1,
"subject": "Task 1: Rewrite failover-drill.sh — standby-victim default + explicit active-victim gap mode",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: Correct the recovery narrative + document the first-seed bootstrap constraint",
"status": "pending",
"status": "completed",
"blockedBy": [
1
]
@@ -18,7 +18,8 @@
{
"id": 3,
"subject": "Task 3: RUN the drill live (both directions) and record measured timings",
"status": "pending",
"status": "completed",
"notes": "The live-drill residual (tracker: 'T2 live-drill deferred') was RESOLVED 2026-08-01 via PLAN-R2-01 T4's live FailoverTimingTests + docker/failover-drill.sh + SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*; recorded in the deferred-work register. Statuses in this manifest flipped from stale 'pending' per 00-MASTER-TRACKER.md by the 2026-08-07 truth sweep.",
"blockedBy": [
1,
2
@@ -27,25 +28,25 @@
{
"id": 4,
"subject": "Task 4: Wire FailoverTimingTests to TwoNodeClusterFixture — measure the ~25s envelope in-process",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: Apply the wonder-app-vd03 appsettings overlay edits (NodeName + AllowSingleNodeCluster, drop phantom seeds)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Complete the install.ps1 recovery actions (sc.exe failureflag) + RUNBOOK recovery step",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 7,
"subject": "Task 7: Correct the factually wrong N2 deferral record in the master tracker",
"status": "pending",
"status": "completed",
"blockedBy": [
5,
6
@@ -54,29 +55,29 @@
{
"id": 8,
"subject": "Task 8: Metrics-staleness for never-reported sites (FirstSeenAt anchor)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 9,
"subject": "Task 9: Purge the stale \"cluster leader\" narration from CentralHealthReportLoop",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 10,
"subject": "Task 10: Generalize the registrar — SingletonRegistrar with an optional role scope",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Route the two site singletons through the registrar — deployment-manager + event-log-handler gain drains",
"status": "pending",
"status": "completed",
"blockedBy": [
10
]
}
],
"lastUpdated": "2026-07-13T03:25:44Z"
"lastUpdated": "2026-08-07T00:00:00Z"
}
archreview/plans/PLAN-R2-02-communication-store-and-forward.md.tasks.json
+15 -15
View File
@@ -4,19 +4,19 @@
{
"id": 1,
"subject": "Task 1: Shared oldest-Up active-node evaluator, reachable from Communication and SiteRuntime",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: Two-node divergence repro — leader≠oldest wipes the delivering node's buffer (failing first)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 3,
"subject": "Task 3: Swap SiteReplicationActor to the shared oldest-Up predicate + Host wires the delivery-gate delegate",
"status": "pending",
"status": "completed",
"blockedBy": [
1,
2
@@ -25,7 +25,7 @@
{
"id": 4,
"subject": "Task 4: Swap SiteCommunicationActor.DefaultIsActiveCheck + Host wiring + doc sync",
"status": "pending",
"status": "completed",
"blockedBy": [
3
]
@@ -33,7 +33,7 @@
{
"id": 5,
"subject": "Task 5: Chunked resync protocol — additive messages, byte-budgeted chunker, active-side chunked answer",
"status": "pending",
"status": "completed",
"blockedBy": [
3
]
@@ -41,7 +41,7 @@
{
"id": 6,
"subject": "Task 6: Standby-side chunk assembly, atomic apply, ack + the N5 race comment",
"status": "pending",
"status": "completed",
"blockedBy": [
5
]
@@ -49,7 +49,7 @@
{
"id": 7,
"subject": "Task 7: Resync delivery confirmation on the active node + telemetry + doc rewrite",
"status": "pending",
"status": "completed",
"blockedBy": [
6
]
@@ -57,25 +57,25 @@
{
"id": 8,
"subject": "Task 8: Publish _sweepTask only when the sweep CAS is won (unclobber the shutdown drain)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 9,
"subject": "Task 9: Eager validation for SweepBatchLimit / SweepTargetParallelism",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 10,
"subject": "Task 10: Coalesce live-alarm delta publishes (bound the per-circuit fan-out) [MUTEX with PLAN-R2-07]",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Aggregator lifecycle — queued re-seed after reconnect + stream-generation stamp",
"status": "pending",
"status": "completed",
"blockedBy": [
10
]
@@ -83,7 +83,7 @@
{
"id": 12,
"subject": "Task 12: Instance-injected ReconnectDelay/StabilityWindow (kill the process-global test seams)",
"status": "pending",
"status": "completed",
"blockedBy": [
11
]
@@ -91,13 +91,13 @@
{
"id": 13,
"subject": "Task 13: Production caller for RemoveSiteAsync — dispose a deleted site's gRPC channels",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 14,
"subject": "Task 14: Documented acceptance — standby-node aggregators + deleted-site viewer behavior",
"status": "pending",
"status": "completed",
"blockedBy": [
4
]
@@ -105,7 +105,7 @@
{
"id": 15,
"subject": "Task 15: Single-endpoint sites can go live [MUTEX with PLAN-R2-07]",
"status": "pending",
"status": "completed",
"blockedBy": [
12
]
archreview/plans/PLAN-R2-03-site-runtime-dcl.md.tasks.json
+7 -7
View File
@@ -1,13 +1,13 @@
{
"planPath": "archreview/plans/PLAN-R2-03-site-runtime-dcl.md",
"tasks": [
{ "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
{ "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "pending", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "pending", "blockedBy": [] },
{ "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6] }
{ "id": 1, "subject": "Task 1: MxGatewayDataConnection — stale event-loop fault on a cancelled token must not signal Disconnected (N1)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: ScriptActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "completed", "blockedBy": [] },
{ "id": 3, "subject": "Task 3: AlarmActor — failure mapping on the expression-eval PipeTo clears _evalInFlight (N2)", "status": "completed", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs (N3)", "status": "completed", "blockedBy": [] },
{ "id": 5, "subject": "Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Wire the compile cache into ScriptCompilationService — deploy gate + PreStart share one compile (N4)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6] }
],
"lastUpdated": "2026-07-13T03:23:02Z"
}
archreview/plans/PLAN-R2-04-data-audit-backbone.md.tasks.json
+13 -13
View File
@@ -1,19 +1,19 @@
{
"planPath": "archreview/plans/PLAN-R2-04-data-audit-backbone.md",
"tasks": [
{ "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "pending", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "pending", "blockedBy": [2, 3] },
{ "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "pending", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "pending", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "pending", "blockedBy": [1, 2] },
{ "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "pending", "blockedBy": [] },
{ "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "pending", "blockedBy": [10] },
{ "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "pending", "blockedBy": [] },
{ "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "pending", "blockedBy": [4, 7, 8, 9, 11, 12] }
{ "id": 1, "subject": "Task 1: Untracked, projected fold fetch (R2)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Rollup watermark seam — GetLatestRollupHourAsync (R1, part 1)", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Slice the rollup backfill into bounded day windows (R1, part 2)", "status": "completed", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Backfill failover fast-path via the rollup watermark + doc correction (R1, part 3)", "status": "completed", "blockedBy": [2, 3] },
{ "id": 5, "subject": "Task 5: Per-metric reduction in the bucketer — sum-per-bucket for Rate series (R3, part 1)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Catalog-driven aggregation at the query-service boundary (R3, part 2)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Truthful trend presentation — chart doc + Component-KpiHistory.md (R3, part 3)", "status": "completed", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Close the catalog metric-literal drift hazard (R4)", "status": "completed", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Classify the failover fold-race failure grain (R5)", "status": "completed", "blockedBy": [1, 2] },
{ "id": 10, "subject": "Task 10: SiteCalls filtered terminal index (migration) (R6, part 1)", "status": "completed", "blockedBy": [] },
{ "id": 11, "subject": "Task 11: Time-sliced SiteCalls terminal purge (R6, part 2)", "status": "completed", "blockedBy": [10] },
{ "id": 12, "subject": "Task 12: Retention-service shutdown no longer surfaces cancellation (R7)", "status": "completed", "blockedBy": [] },
{ "id": 13, "subject": "Final verification: solution build + targeted suites this plan touched (infra up for MSSQL SkippableFacts)", "status": "completed", "blockedBy": [4, 7, 8, 9, 11, 12] }
],
"lastUpdated": "2026-07-13T03:25:01Z"
}
archreview/plans/PLAN-R2-05-templates-deployment-transport.md.tasks.json
+9 -9
View File
@@ -1,15 +1,15 @@
{
"planPath": "archreview/plans/PLAN-R2-05-templates-deployment-transport.md",
"tasks": [
{ "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "pending", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "pending", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "pending", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "pending", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "pending", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] }
{ "id": 1, "subject": "Task 1: Full violation/error lists in the trigger-expression syntax check (N2)", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Add a globals-surface discriminator to the verdict-cache key (N1 part 1)", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: Cache Expression-trigger verdicts under the trigger surface key (N1 part 2)", "status": "completed", "blockedBy": [2] },
{ "id": 4, "subject": "Task 4: Skip the trigger syntax check on read-only staleness/comparison paths (N1 part 3)", "status": "completed", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Publish ScriptArtifactsChanged for Add resolutions too (N3)", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: Trust-gate instance alarm-override trigger expressions (N5)", "status": "completed", "blockedBy": [5] },
{ "id": 7, "subject": "Task 7: Warn when import persists overrides on locked template members (N4)", "status": "completed", "blockedBy": [6] },
{ "id": 8, "subject": "Task 8: Validate MaxConcurrentImportSessions at startup (N6)", "status": "completed", "blockedBy": [] },
{ "id": 9, "subject": "Task 9: Design-doc sync sweep", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8] }
],
"lastUpdated": "2026-07-13T03:23:07Z"
}
archreview/plans/PLAN-R2-06-edge-integrations.md.tasks.json
+6 -6
View File
@@ -1,12 +1,12 @@
{
"planPath": "archreview/plans/PLAN-R2-06-edge-integrations.md",
"tasks": [
{ "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "pending", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "pending", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "pending", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "pending", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "pending", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "pending", "blockedBy": [] }
{ "id": 1, "subject": "Task 1: ScriptArtifactChangeSubscriber — wire the Inbound API as the bus's ApiMethod consumer", "status": "completed", "blockedBy": [] },
{ "id": 2, "subject": "Task 2: Truth sweep — tracker / contract-doc / CLAUDE.md claims about the bus consumer", "status": "completed", "blockedBy": [1] },
{ "id": 3, "subject": "Task 3: DeliverBufferedAsync parks deterministic ArgumentException failures (path template / verb)", "status": "completed", "blockedBy": [] },
{ "id": 4, "subject": "Task 4: Honor the declared response charset in ReadBodyBoundedAsync", "status": "completed", "blockedBy": [3] },
{ "id": 5, "subject": "Task 5: Typed SITE_UNREACHABLE — AskTimeoutException classification replaces message sniffing", "status": "completed", "blockedBy": [] },
{ "id": 6, "subject": "Task 6: 415 guard covers chunked (no Content-Length) bodies", "status": "completed", "blockedBy": [] }
],
"lastUpdated": "2026-07-13T03:23:07Z"
}
archreview/plans/PLAN-R2-07-ui-management-security.md.tasks.json
+12 -12
View File
@@ -4,19 +4,19 @@
{
"id": 1,
"subject": "Task 1: Route the DebugStreamHub LDAP bind through ManagementAuthenticator (throttled) + fix the false doc claim",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 2,
"subject": "Task 2: ForwardedHeadersSetup — trusted-proxy client-IP resolution for the throttle keys",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 3,
"subject": "Task 3: Ship the ForwardedHeaders config to the Traefik topologies + document the lockout-DoS trade-off",
"status": "pending",
"status": "completed",
"blockedBy": [
2
]
@@ -24,19 +24,19 @@
{
"id": 4,
"subject": "Task 4: Enforce site scope on secured-write submit / approve / reject",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: ISecuredWriteRepository — additive permitted-sites filter for scoped listing",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Scope-filter HandleListSecuredWrites + amend Component-Security.md:141 + matrix verification",
"status": "pending",
"status": "completed",
"blockedBy": [
4,
5
@@ -45,13 +45,13 @@
{
"id": 7,
"subject": "Task 7: AlarmSummary.RefreshAsync stale-site guard — never display cross-site data",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 8,
"subject": "Task 8: While live, the poll updates only _notReporting — never regresses live rows",
"status": "pending",
"status": "completed",
"blockedBy": [
7
]
@@ -59,7 +59,7 @@
{
"id": 9,
"subject": "Task 9: House disposal guard on the live callback",
"status": "pending",
"status": "completed",
"blockedBy": [
8
]
@@ -67,19 +67,19 @@
{
"id": 10,
"subject": "Task 10: Un-stick IsLive — deathwatch resets liveness when the aggregator terminates (MUTEX with PLAN-R2-02 SiteAlarmLiveCacheService task)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Amortize LoginThrottle.Prune off the failure hot path",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 12,
"subject": "Task 12: Lock in the scrubber's fragment coverage + document the array-merge limitation",
"status": "pending",
"status": "completed",
"blockedBy": []
}
],
archreview/plans/PLAN-R2-08-conventions-hygiene.md.tasks.json
+20 -16
View File
@@ -5,80 +5,84 @@
"id": 1,
"subject": "Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — needs-user",
"status": "pending",
"blockedBy": []
"blockedBy": [],
"notes": "Still open per 00-MASTER-TRACKER.md (human-gated residual; needs-user)."
},
{
"id": 2,
"subject": "Task 2: Delete test.txt and triage the root credential files — needs-user",
"status": "pending",
"blockedBy": [1]
"blockedBy": [1],
"notes": "Still open per 00-MASTER-TRACKER.md (human-gated residual; needs-user)."
},
{
"id": 3,
"subject": "Task 3: Root secret-capture guards — .gitignore patterns + pre-commit secret scan",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 4,
"subject": "Task 4: CHANGELOG.md — correct the false role claim and refresh to reality",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 5,
"subject": "Task 5: Options validation — AuditLog site sub-options (SiteWriter, SiteTelemetry, SiteRetention)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 6,
"subject": "Task 6: Options validation — AuditLog central sub-options (PartitionMaintenance, Purge, Reconciliation)",
"status": "pending",
"status": "completed",
"blockedBy": [5]
},
{
"id": 7,
"subject": "Task 7: Options validation — Host NodeOptions / DatabaseOptions / LoggingOptions (empty NodeName fails fast)",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 8,
"subject": "Task 8: OperationTrackingOptions binding + the missing site IOperationTrackingStore registration (verify-then-fix)",
"status": "pending",
"status": "completed",
"blockedBy": [7]
},
{
"id": 9,
"subject": "Task 9: NF8 — canonicalize the Communication/DataConnection section names, drop the duplicate Host bindings",
"status": "pending",
"status": "completed",
"blockedBy": [8]
},
{
"id": 10,
"subject": "Task 10: Re-consolidate deferral tracking into the canonical register",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 11,
"subject": "Task 11: Delete the drifted root deferred.md snapshot — needs-user",
"status": "pending",
"blockedBy": [10]
"status": "completed",
"blockedBy": [10],
"notes": "Completed 2026-08-07 (truth sweep): deferred.md was in fact TRACKED (the plan's 'untracked' premise was wrong), so it was removed via git rm."
},
{
"id": 12,
"subject": "Task 12: Fix the stale Transport area comment in EntitySerializer.FromBundleContent",
"status": "pending",
"status": "completed",
"blockedBy": []
},
{
"id": 13,
"subject": "Task 13: Commit the orphaned MES plan doc; remove the generated root reports — needs-user (deletions only)",
"status": "pending",
"blockedBy": []
"status": "completed",
"blockedBy": [],
"notes": "Completed 2026-08-07 (truth sweep): ScadaBridge-docs-issues.md + ScadaBridge-docs-fixed.md were in fact TRACKED and removed via git rm; the MES plan doc was already committed earlier (3c9b101d). Other tasks in this manifest flipped to completed per 00-MASTER-TRACKER.md (round 2 merged to main @ 1930f19b, 2026-07-13)."
}
],
"lastUpdated": "2026-07-13T03:30:22Z"
"lastUpdated": "2026-08-07T00:00:00Z"
}
deferred.md
-44
View File
@@ -1,44 +0,0 @@
# Remaining Deferred Work
Source: `docs/plans/2026-07-08-deferred-work-register.md` (snapshot 2026-07-10).
Everything in the register's "Fix-now" table is already landed via the archreview
plans; what's left are the intentional deferrals below.
## Product / roadmap-locked (revisit needs a decision or a trigger event)
| # | Item | Why deferred | Revisit trigger |
|---|------|--------------|-----------------|
| 8 | Hash-chain tamper evidence (CLI `verify-chain` is a no-op stub) | v1.x by locked decision — append-only DB roles are the current control | A compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (endpoint returns `501`) | v1.x — the `501` + CLI messaging are honest, not broken | AuditLog partition volume nears the retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | Broadcast-to-both-nodes already covers HA | A governance/audit requirement for trust decisions |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | v1 manifest hash + AES-GCM held sufficient | A non-repudiation requirement across orgs |
| 17 | Unified notifications + site-calls outbox page | Explicit M9 decision to keep two pages | Operator confusion reports |
| 18 | Folder drag-drop | Permanently closed — menu reorder shipped instead | — (closed) |
## Scale / YAGNI (deferred until load justifies it)
| # | Item | Why deferred | Revisit trigger | Status |
|---|------|--------------|-----------------|--------|
| 10 | Aggregated live alarm stream for Alarm Summary | Snapshot fan-out is acceptable at current instance counts | Latency complaints or >~50 instances/site | ✅ **SHIPPED + MERGED to main 2026-07-10** (`8c888f13`, plan `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`, T1–T8). Transient per-site in-memory live cache (`ISiteAlarmLiveCache`/`SiteAlarmAggregatorActor`) seeded by snapshot fan-out + additive `SubscribeSite` alarm-only gRPC stream; live-cache-driven Alarm Summary with 15s poll fallback; `[PERM]` no-central-store honored (code-reviewer-confirmed); validated options + telemetry; end-to-end trace. Register row moved to Resolved. |
| 22 | KPI history hourly rollups | 90-day retention already bounds the table | `KpiSample` query latency on dashboards | ✅ **SHIPPED + MERGED to main 2026-07-10** (`8c888f13`, plan `docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8). Separate `KpiRollupHourly` table (migration `20260710153953`), recorder hourly fold w/ failover-safe lookback re-fold + idempotent upsert, per-metric gauge-vs-rate aggregation, range-threshold query routing (`RollupThresholdHours` 168h), longer rollup retention (365 ≥ 90) + dual purge, one-shot backfill, and 30 d/90 d window buttons. Register row moved to Resolved. |
## Low-priority polish (near-complete, small remainder)
| # | Item | Why deferred | Revisit trigger |
|---|------|--------------|-----------------|
| 12 | Native-alarm-source-override CSV import — Central UI upload button only | CLI + Management API + parser shipped 2026-07-10; the Blazor upload affordance is the only piece left, and it's pure polish (needs a live Blazor smoke) | First request to bulk-import native sources from the UI instead of the CLI |
## New deferrals from review 08 (engineering debt, no defect)
| Item | Why deferred | Revisit trigger |
|------|--------------|-----------------|
| Communication → HealthMonitoring layering inversion | Moving the interface + `SiteHealthState` to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to `ICentralHealthAggregator` |
| Reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Full StyleGuide-conformant docs are substantial; README claim was scoped instead (PLAN-08 T10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests, DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | Needs the PLAN-01 two-node rig; placeholder harness already shipped (PLAN-08 T8) | PLAN-01 rig landing |
---
Summary: 12 open deferrals (13th, folder drag-drop #18, is permanently closed).
None are currently actionable without a triggering event or product decision —
except row #12's UI upload button, whose CLI/API/parser core already shipped.
docker-env2/central-node-a/appsettings.Central.json
+1 -1
View File
@@ -11,7 +11,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/central-node-b/appsettings.Central.json
+3 -3
View File
@@ -8,10 +8,10 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
docker-env2/docker-compose.yml
+8
View File
@@ -16,6 +16,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports:
- "9101:5000" # Web UI + Inbound API
- "9111:8081" # Akka remoting
@@ -43,6 +47,10 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
ScadaBridge__Communication__SitePsks__site-x: "dev-grpc-psk-docker-env2-site-x"
ports:
- "9102:5000" # Web UI + Inbound API
- "9112:8081" # Akka remoting
docker-env2/site-x-node-a/appsettings.Site.json
+20 -6
View File
@@ -13,13 +13,17 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -29,13 +33,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docker-env2/site-x-node-b/appsettings.Site.json
+22 -8
View File
@@ -10,16 +10,20 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -29,13 +33,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-env2-site-x",
"CentralGrpcEndpoints": [
"http://scadabridge-env2-central-a:8083",
"http://scadabridge-env2-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docker/README.md
+66 -12
View File
@@ -120,6 +120,31 @@ docker/
└── logs/
```
## gRPC control-plane keys (dev)
The site gRPC service (`SiteStreamService` on 8083 — live subscriptions, audit pull,
cached-telemetry ingest) is gated by a preshared key, and the gate is **fail-closed**: a site node
with no key refuses every call, and `StartupValidator` refuses to boot it at all. So the rig
carries dev keys, one per site:
| Where | Setting | Value |
|---|---|---|
| `site-{a,b,c}-node-*/appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | `dev-grpc-psk-docker-site-{a,b,c}` |
| `docker-compose.yml`, both central nodes | `ScadaBridge__Communication__SitePsks__site-{a,b,c}` | same value |
Both nodes of a pair carry the same key; each site's key is different from the others'. The
central half lives in compose env rather than the mounted `appsettings.Central.json`, which by
convention holds no plaintext credentials. Production uses `${secret:SB-GRPC-PSK-<siteId>}` on
the site and the matching secret in central's store — see
[`docs/deployment/topology-guide.md`](../docs/deployment/topology-guide.md).
**These are not real secrets and are committed deliberately**, exactly like the LocalDb sync key
(`dev-site-a-localdb-sync-key`) beside them. The two are separate keys on purpose: the LocalDb one
authenticates the *pair partner* for database replication, not central.
If you add a site to the rig, add its key in both places or its streams will fail with
`PermissionDenied`.
## Commands
### Initial Setup
@@ -273,29 +298,43 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
### Automated failover drill (`failover-drill.sh`)
```bash
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER
```
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered.
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** The survivor must **take over while the victim is still down**: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its `/health/active` goes 200. PASS = survivor active within `TIMEOUT_S`, then Traefik routing to it. (Under the pre-2026-07-21 `keep-oldest` strategy this direction was a proven total outage — the younger survivor took `DownReachable` and downed itself, because Akka's `down-if-alone` only rescues a side with ≥ 2 members.)
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
> **Observed results** (plan R2-01 T3):
**Seed-node ordering — every node lists ITSELF first (decision 2026-07-22).** Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node's own address; every other node runs `JoinSeedNodeProcess`, which retries `InitJoin` forever and can never form a cluster. Each shipped node config therefore lists itself first and its partner second (`docker/central-node-b/appsettings.Central.json` leads with `scadabridge-central-b`), and `StartupValidator` fails the boot if that ordering is ever broken. This closes the former **registered outage gap**, where a lone cold-starting `central-b` (with `central-a` down) never came `Up` and recovery was operator-driven.
Self-first ordering is safe, and the three interesting cases are covered by `SelfFirstSeedBootstrapTests` (real in-process clusters at production failure-detection timings):
| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Forms alone in ~5s (`seed-node-timeout`) — operational, unattended |
| Restart into a **live** peer | `InitJoinAck` answers, node rejoins; never islands |
| Both cold-start simultaneously (mutually reachable) | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
> An earlier revision of this README claimed the repo deliberately avoided self-first ordering because simultaneous cold start would produce "two one-node clusters that never merge". That is **not** what happens while the nodes are mutually reachable — the handshake converges them (measured, row 3 above). Only a genuine boot-time *partition* splits them, which is the same class `auto-down` already accepts.
> **Rejected alternative — an external self-form timer.** A watchdog that waits N seconds for membership and then calls `Cluster.Join(SelfAddress)` was implemented and discarded: it cannot see Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation; a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address — a **permanent** split (measured: still split after 90s). Akka's own first-seed process has no such race because it *is* part of the handshake.
> **Observed results** (auto-down decision verification):
>
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts.
> **Run 2026-07-21** against a freshly-deployed cluster with `SplitBrainResolverStrategy: auto-down` (first drill: `active=central-a`). Both directions recovered.
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. |
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. |
> | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
> | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
>
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**.
> Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
### Central Failover
@@ -313,6 +352,14 @@ open http://localhost:9002
docker start scadabridge-central-a
```
**Manual failover from the UI (admin-only).** Instead of stopping a container, an Administrator can trigger a planned role swap from the **Trigger failover** button on the central-cluster card at `/monitoring/health` (via Traefik, `http://localhost:9000`). The active (oldest Up) node leaves the cluster **gracefully**, so singletons hand over rather than being killed; the node then restarts under `restart: unless-stopped` and rejoins as the standby.
- The button is disabled when the pair has no online standby — the same guard is re-enforced server-side, since failing over a lone node is an outage, not a failover.
- Triggering it **disconnects the page you clicked it on**: Traefik routes the UI to the active node, which is the node being restarted. The page reconnects against the new active node.
- Each invocation writes one `Cluster` / `ManualFailover` row to `dbo.AuditLog` naming the admin and the target address, written before the Leave is issued.
To verify on the rig: press the button, watch `central-a` restart and `central-b`'s badge flip to Primary, then confirm the audit row landed.
### Site Failover
```bash
@@ -329,3 +376,10 @@ docker start scadabridge-site-a-a
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.
docker/central-node-a/appsettings.Central.json
+9 -2
View File
@@ -11,11 +11,18 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
docker/central-node-b/appsettings.Central.json
+11 -4
View File
@@ -8,14 +8,21 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
"akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-a:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
docker/docker-compose.yml
+115
View File
@@ -1,12 +1,81 @@
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
#
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
# the wire, so every participating node must resolve the SAME KEK.
#
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
# site-b and site-c are deliberately left without it, so the default-OFF posture is
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
# pair. A node with no Secrets__* override keeps the shipped appsettings default
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
#
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
# since resolving one is what the hub exists to make possible.
x-secrets-hub-env: &secrets-hub-env
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
# kek_id mismatch that reads like corruption but is a deployment error.
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
# The appsettings default is the relative "scadabridge-secrets.db", which resolves to
# /app — inside the image's writable layer, so it is destroyed by any container
# recreate and unreachable from the host. /app/data is the node's own mounted volume
# (the one LocalDb already uses on sites; added to the central pair for this).
Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__Mode: "Grpc"
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
# hub and every follower; an unset token is a startup failure on both halves.
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
# so the two hub instances serve identical manifests by construction — that is what makes
# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
# reach the store that would resolve it (bootstrap circularity; registration rejects it).
# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
x-secrets-hub-central-env: &secrets-hub-central-env
Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# Site half of the same section. The hub client dials Endpoint first and fails over, per
# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
# above — never list endpoints backed by independent stores: failing over to an emptier
# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
# exists to prevent.
x-secrets-hub-site-env: &secrets-hub-site-env
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
services:
central-a:
image: scadabridge:latest
# An init process (tini) as PID 1, so a crashed dotnet process actually dies.
# Without it dotnet IS PID 1, Linux ignores the SIGABRT the runtime's crash path
# raises against PID 1, and any unhandled boot exception left the container
# `running` with the main thread spinning at 100% CPU — restart policy never
# fired (ScadaBridge#34). Belt to Program.cs's UnhandledException handler, which
# covers managed exceptions but not FailFast/runtime-internal aborts.
init: true
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-a
environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
@@ -27,11 +96,26 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9001:5000" # Web UI + Inbound API
- "9011:8081" # Akka remoting (host access for CLI/debugging)
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-a/data:/app/data
- ./central-node-a/logs:/app/logs
networks:
- scadabridge-net
@@ -39,12 +123,16 @@ services:
central-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-central-b
environment:
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
# (anchors at the top of this file).
<<: [*secrets-hub-env, *secrets-hub-central-env]
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
@@ -65,11 +153,26 @@ services:
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
ScadaBridge__Security__JwtSigningKey: "scadabridge-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared keys, one per site — NOT real secrets.
# Central verifies/presents these; each site node carries the same value as
# ScadaBridge:Communication:GrpcPsk in its mounted appsettings.Site.json. Kept as
# env overrides (not in the mounted central appsettings) so that file stays free of
# plaintext credentials. Production instead seeds SB-GRPC-PSK-<siteId> into the
# secret store, which is also the only source that can serve a site added at runtime.
ScadaBridge__Communication__SitePsks__site-a: "dev-grpc-psk-docker-site-a"
ScadaBridge__Communication__SitePsks__site-b: "dev-grpc-psk-docker-site-b"
ScadaBridge__Communication__SitePsks__site-c: "dev-grpc-psk-docker-site-c"
ports:
- "9002:5000" # Web UI + Inbound API
- "9012:8081" # Akka remoting
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
# Originally added for the gRPC secrets hub's local SQLite store; since the
# central store moved to the shared SQL-Server database (Secrets 0.5.0,
# scadaproj#4) the scadabridge-secrets.db here is a pre-0.5.0 residue, but the
# volume is still needed (inbound-api-keys.sqlite lives on it).
- ./central-node-b/data:/app/data
- ./central-node-b/logs:/app/logs
networks:
- scadabridge-net
@@ -77,12 +180,16 @@ services:
site-a-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-a
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9021:8082" # Akka remoting (host access for debugging)
@@ -97,12 +204,16 @@ services:
site-a-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
stop_grace_period: 30s
container_name: scadabridge-site-a-b
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9022:8082" # Akka remoting
@@ -117,6 +228,7 @@ services:
site-b-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -137,6 +249,7 @@ services:
site-b-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -157,6 +270,7 @@ services:
site-c-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -177,6 +291,7 @@ services:
site-c-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
docker/failover-drill.sh
+52 -39
View File
@@ -1,33 +1,33 @@
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
# downing strategy (availability-first): the leader among the REACHABLE members
# downs the unreachable peer after StableAfter, so a hard crash of EITHER
# central node — the active/oldest included — fails over to the survivor. The
# accepted trade (made explicitly by the owner) is dual-active during a real
# network partition. Both drill directions therefore expect RECOVERY:
#
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node
# keeps its singletons, and Traefik routing never goes dark. PASS = the
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
# failure detection + 15s stable-after) while /health/active stays up.
# The active node is untouched: expect zero /health/active routing blips
# and member removal on the survivor within ~25s (10s failure detection +
# 15s auto-down-unreachable-after).
#
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
# cannot help — the alone-oldest is dead and cannot down itself), and the
# self-downed survivor cannot re-form a cluster alone unless it is the
# FIRST seed (both nodes list central-a first; only the first seed may
# self-join). This mode measures the dark window and PASSes only when
# central recovers AFTER the victim container is restarted. It exists to
# make the registered gap observable — not to pretend it is covered.
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
# the singletons, and /health/active goes 200 on the survivor WHILE THE
# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
# was a total outage — the younger survivor downed ITSELF, verified live;
# Akka's down-if-alone only rescues a side with >= 2 members.)
#
# Both modes finish by restarting the victim and confirming it rejoins as a
# fresh incarnation (standby).
set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
@@ -35,6 +35,7 @@ active_container() {
else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
case "$DRILL_MODE" in
standby|active) ;;
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi
SURVIVOR_PORT=$(port_of "$SURVIVOR")
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
BLIPS=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break
fi
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
DARK_STREAK=0
echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
while true; do
ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
if (( DARK_STREAK >= 10 )); then
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
break
fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
break
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Confirming Traefik routes to the new active node..."
TR_START=$(date +%s)
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
if (( $(date +%s) - TR_START > 60 )); then
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
fi
echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
VICTIM_PORT=$(port_of "$VICTIM")
while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
break
fi
if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
exit 1
fi
sleep 1
done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
docker/regen-proto.sh
+88 -49
View File
@@ -1,92 +1,131 @@
#!/usr/bin/env bash
#
# Regenerates the gRPC C# files from sitestream.proto.
# Regenerates the gRPC C# files from the Communication project's .proto files.
#
# Background: protoc (linux/arm64) segfaults inside our Docker build container
# (Grpc.Tools 2.71.0). As a workaround the generated Sitestream.cs +
# SitestreamGrpc.cs are checked into src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/
# and the Protobuf ItemGroup in the .csproj is commented out — Docker just
# compiles the checked-in C# files.
# (Grpc.Tools). As a workaround the generated C# is checked into
# src/ZB.MOM.WW.ScadaBridge.Communication/ — SiteStreamGrpc/ for sitestream.proto,
# CentralControlGrpc/ for central_control.proto, and SiteCommandGrpc/ for
# site_command.proto — and the Protobuf ItemGroup in the .csproj is commented out,
# so Docker just compiles the checked-in files.
#
# Run this script ON YOUR DEV MACHINE whenever Protos/sitestream.proto changes:
# Run this script ON YOUR DEV MACHINE whenever a .proto changes:
#
# 1. Temporarily uncomments the Protobuf ItemGroup so Grpc.Tools runs.
# 2. dotnet build (regen writes fresh files to obj/).
# 3. Copies the regenerated files back into SiteStreamGrpc/.
# 4. Re-comments the Protobuf ItemGroup so Docker builds stay safe.
# docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all] (default: all)
#
# 1. Injects a Protobuf ItemGroup for the selected proto(s) so Grpc.Tools runs.
# 2. Deletes the stale checked-in C# so a failed regen is obvious.
# 3. dotnet build (regen writes fresh files to obj/).
# 4. Copies the regenerated files back into the source tree.
# 5. Restores the original csproj so no active Protobuf item is left behind.
#
# Only the SELECTED protos get a Protobuf item. Enabling one whose generated C#
# is still checked in would define every generated type twice, which is why the
# per-proto selection exists. central_control.proto imports sitestream.proto,
# but protoc resolves that from the project-relative path — the import needs no
# Protobuf item of its own.
#
# Once we move to a Dockerfile base image that ships a working linux/arm64
# protoc, this script can be retired and Docker can regen the proto on every
# protoc, this script can be retired and Docker can regen the protos on every
# build like every other normal .NET project.
set -euo pipefail
TARGET="${1:-all}"
case "$TARGET" in
sitestream|centralcontrol|sitecommand|all) ;;
*) echo "usage: $0 [sitestream|centralcontrol|sitecommand|all]" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
COMM_DIR="$REPO_ROOT/src/ZB.MOM.WW.ScadaBridge.Communication"
CSPROJ="$COMM_DIR/ZB.MOM.WW.ScadaBridge.Communication.csproj"
GEN_DIR="$COMM_DIR/SiteStreamGrpc"
GEN="$COMM_DIR/obj/Debug/net10.0/Protos"
echo "=== Regenerating gRPC files from sitestream.proto ==="
echo "=== Regenerating gRPC files ($TARGET) ==="
if [[ ! -f "$CSPROJ" ]]; then
echo "ERROR: csproj not found at $CSPROJ" >&2
exit 1
fi
# Backup so we can always restore the comment state on failure.
# Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
# 1. Uncomment the Protobuf ItemGroup (strip the surrounding <!-- ... --> wrapper).
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
# Find the commented Protobuf block and unwrap it.
new = re.sub(
r"<!--\s*\n(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)\s*\n\s*-->",
r"\1",
src,
count=1,
)
if new == src:
raise SystemExit("Couldn't find commented Protobuf ItemGroup to enable.")
p.write_text(new)
# 1. Inject an ItemGroup holding just the selected protos, immediately before
# the closing </Project>. The documented commented-out block is left alone.
python3 - "$CSPROJ" "$TARGET" <<'PY'
import pathlib, sys
csproj, target = pathlib.Path(sys.argv[1]), sys.argv[2]
protos = []
if target in ("sitestream", "all"):
protos.append("sitestream.proto")
if target in ("centralcontrol", "all"):
protos.append("central_control.proto")
if target in ("sitecommand", "all"):
protos.append("site_command.proto")
items = "\n".join(
f' <Protobuf Include="Protos\\{p}" GrpcServices="Both" />' for p in protos)
block = f" <ItemGroup>\n{items}\n </ItemGroup>\n\n</Project>"
src = csproj.read_text()
if "</Project>" not in src:
raise SystemExit("Couldn't find </Project> to inject the Protobuf ItemGroup before.")
csproj.write_text(src.replace("</Project>", block, 1))
PY
# 2. Delete the stale files so any failure to regen is obvious.
rm -f "$GEN_DIR/Sitestream.cs" "$GEN_DIR/SitestreamGrpc.cs"
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteStreamGrpc/Sitestream.cs" "$COMM_DIR/SiteStreamGrpc/SitestreamGrpc.cs"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/CentralControlGrpc/CentralControl.cs" \
"$COMM_DIR/CentralControlGrpc/CentralControlGrpc.cs"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
rm -f "$COMM_DIR/SiteCommandGrpc/SiteCommand.cs" \
"$COMM_DIR/SiteCommandGrpc/SiteCommandGrpc.cs"
fi
# 3. Regenerate by building.
echo "Building Communication project (regen)..."
dotnet build "$CSPROJ" --nologo -v minimal | tail -5
# 4. Copy generated files back into the source tree.
mkdir -p "$GEN_DIR"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/Sitestream.cs" "$GEN_DIR/Sitestream.cs"
cp "$COMM_DIR/obj/Debug/net10.0/Protos/SitestreamGrpc.cs" "$GEN_DIR/SitestreamGrpc.cs"
echo "Copied regenerated files to $GEN_DIR/"
# 5. Re-comment the Protobuf ItemGroup so Docker builds keep working.
python3 - <<PY
import re, pathlib
p = pathlib.Path("$CSPROJ")
src = p.read_text()
new = re.sub(
r"(\s*<ItemGroup>\s*\n\s*<Protobuf [^>]*/>\s*\n\s*</ItemGroup>)",
r"\n <!--\1\n -->",
src,
count=1,
)
p.write_text(new)
PY
if [[ "$TARGET" == "sitestream" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteStreamGrpc"
cp "$GEN/Sitestream.cs" "$GEN/SitestreamGrpc.cs" "$COMM_DIR/SiteStreamGrpc/"
echo "Copied regenerated files to SiteStreamGrpc/"
fi
if [[ "$TARGET" == "centralcontrol" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/CentralControlGrpc"
cp "$GEN/CentralControl.cs" "$GEN/CentralControlGrpc.cs" "$COMM_DIR/CentralControlGrpc/"
echo "Copied regenerated files to CentralControlGrpc/"
fi
if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
mkdir -p "$COMM_DIR/SiteCommandGrpc"
cp "$GEN/SiteCommand.cs" "$GEN/SiteCommandGrpc.cs" "$COMM_DIR/SiteCommandGrpc/"
echo "Copied regenerated files to SiteCommandGrpc/"
fi
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP"
trap - ERR
echo ""
echo "Done. Review and commit:"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/Protos/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/"
echo " git diff src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/"
echo " git diff -- src/ZB.MOM.WW.ScadaBridge.Communication/*.csproj # must be EMPTY"
docker/seed-sites.sh
+9 -4
View File
@@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do
done
echo ""
echo "Seeding LDAP group mappings (Design + Deployment)..."
echo "Seeding LDAP group mappings (Designer + Deployer)..."
# SecurityConfiguration.HasData declares 4 mappings but the InitialSchema
# migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts
# with multi-role getting Admin only -- no Design and no Deployment access.
@@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-d ScadaBridgeConfig -Q "
SET IDENTITY_INSERT LdapGroupMappings ON;
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment');
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer');
-- Role strings MUST match the canonical vocabulary in
-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer').
-- These rows previously carried the pre-rename 'Design' / 'Deployment', which
-- authorized nothing: every Designer/Deployer-gated management command failed
-- UNAUTHORIZED on a freshly reseeded rig.
SET IDENTITY_INSERT LdapGroupMappings OFF;
"
docker/site-a-node-a/appsettings.Site.json
+46 -8
View File
@@ -14,13 +14,24 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
@@ -80,7 +101,24 @@
// pre-host secret expander.
"Replication": {
"PeerAddress": "http://scadabridge-site-a-b:8083",
"ApiKey": "dev-site-a-localdb-sync-key"
"ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
}
}
docker/site-a-node-b/appsettings.Site.json
+48 -10
View File
@@ -11,16 +11,27 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-a",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
@@ -73,7 +94,24 @@
// fail-closed, so a typo here does not degrade to unauthenticated replication;
// it rejects every stream and the pair silently stops converging.
"Replication": {
"ApiKey": "dev-site-a-localdb-sync-key"
"ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
}
}
docker/site-b-node-a/appsettings.Site.json
+28 -7
View File
@@ -14,13 +14,24 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docker/site-b-node-b/appsettings.Site.json
+30 -9
View File
@@ -11,16 +11,27 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-b",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docker/site-c-node-a/appsettings.Site.json
+28 -7
View File
@@ -14,13 +14,24 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docker/site-c-node-b/appsettings.Site.json
+30 -9
View File
@@ -11,16 +11,27 @@
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
"BootstrapGuard": {
"Enabled": true,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db"
},
"DataConnection": {
@@ -30,13 +41,23 @@
"SeedReadTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db",
"ReplicationEnabled": true
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
},
"Communication": {
"CentralContactPoints": [
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
// DEV-ONLY control-plane preshared key — NOT a real secret. Must be
// IDENTICAL on both nodes of the pair and match the central-side entry in
// ScadaBridge__Communication__SitePsks__<siteId> (docker-compose.yml).
// Production supplies this as ${secret:SB-GRPC-PSK-<siteId>}. Without it the
// node fails StartupValidator: the gate is fail-closed, so an unset key would
// refuse every SiteStream call while the node still looked healthy.
"GrpcPsk": "dev-grpc-psk-docker-site-c",
"CentralGrpcEndpoints": [
"http://scadabridge-central-a:8083",
"http://scadabridge-central-b:8083"
],
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
docs/components/ClusterInfrastructure.md
+96 -61
View File
@@ -1,23 +1,33 @@
# Cluster Infrastructure
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, split-brain resolution, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.
## Overview
Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project:
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, split-brain settings), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons.
This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.
Both central and site clusters run this same topology: two nodes, one active (cluster leader), one standby, with automatic failover and no manual intervention required for dual-node recovery.
Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery.
## Key Concepts
### Active/standby via cluster leadership
### One `ActorSystem` name for every cluster
Akka.NET cluster leadership determines which node is "active". The cluster leader is the oldest node in the cluster, as tracked by the keep-oldest split-brain resolver. `ActiveNodeGate` (in the Host) exposes `IsActiveNode` by checking whether `cluster.SelfMember.Status == MemberStatus.Up` and `cluster.State.Leader == cluster.SelfAddress`. Cluster singletons — which run on the oldest `Up` member — automatically migrate to the surviving node on failover.
Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system.
### Active/standby is the oldest `Up` member — never the cluster leader
A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`:
- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`).
- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call.
- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`).
Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.
### Configuration contract vs. bootstrap split
@@ -33,22 +43,26 @@ Cluster Infrastructure provides the hosting platform; each singleton is owned an
`AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits three additional stanzas: `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).
The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`):
```csharp
// Abbreviated — see AkkaHostedService.BuildHocon for the full method.
public static string BuildHocon(
NodeOptions nodeOptions,
ClusterOptions clusterOptions,
IEnumerable<string> roles,
TimeSpan transportHeartbeat,
TimeSpan transportFailure)
{
var seedNodesStr = string.Join(",",
clusterOptions.SeedNodes.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
var downingBlock = string.Equals(
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}";
return $@"
return $@"
audit-telemetry-dispatcher {{
type = ForkJoinDispatcher
throughput = 100
@@ -66,13 +80,7 @@ akka {{
seed-nodes = [{seedNodesStr}]
roles = [{rolesStr}]
min-nr-of-members = {clusterOptions.MinNrOfMembers}
split-brain-resolver {{
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
stable-after = {DurationHocon(clusterOptions.StableAfter)}
keep-oldest {{
down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
}}
}}
{downingBlock}
failure-detector {{
heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
@@ -83,23 +91,35 @@ akka {{
run-by-clr-shutdown-hook = on
}}
}}";
}
```
A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.
The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.
### Split-brain resolution
Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.
The keep-oldest strategy is the only strategy `ClusterOptionsValidator` permits for ScadaBridge's two-node clusters. Quorum strategies (`keep-majority`, `static-quorum`) cannot distinguish a crash from a partition with two nodes — both sides would be below quorum and both would shut down. Keep-oldest with `down-if-alone = on` ensures at most one node runs the cluster at any time:
### Downing strategy (auto-down — availability-first)
- On a network partition, the older node stays active; the younger node downs itself.
- If the oldest node finds itself alone (no reachable members), it downs itself rather than running in isolation. Without `down-if-alone`, the oldest node could run as a single-node cluster while the younger node forms its own — producing two live clusters with divergent singleton state.
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.
`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node.
### Downed-node recovery
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
### Failure detection and failover timeline
Detection uses two independent Akka heartbeat channels:
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers `Unreachable` events that the split-brain resolver acts on.
- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on.
- **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`.
With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds:
@@ -107,28 +127,33 @@ With the defaults in `ClusterOptions`, the total failover budget is approximatel
| Phase | Duration | Source |
|-------|----------|--------|
| Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` |
| Split-brain stable-after | 15 s | `ClusterOptions.StableAfter` |
| Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` |
| Singleton restart | < 1 s | Actor `PreStart` |
The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.
### Graceful shutdown and singleton handover
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. `SiteCallAuditActor` has an explicit graceful-stop task registered on `PhaseClusterLeave` with a 10-second timeout to drain any in-flight EF Core upsert before handover opens:
When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.
Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:
```csharp
siteCallAuditShutdown.AddTask(
// SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
"drain-site-call-audit-singleton",
$"drain-{name}-singleton",
async () =>
{
try
{
await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10));
await manager.GracefulStop(timeout);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"SiteCallAudit singleton did not drain within the graceful-stop "
+ "timeout; falling through to PoisonPill handover");
logger.LogWarning(ex,
"{Singleton} singleton did not drain within the graceful-stop timeout; "
+ "falling through to PoisonPill handover", name);
}
return Akka.Done.Instance;
});
@@ -136,25 +161,29 @@ siteCallAuditShutdown.AddTask(
### Cluster roles and singleton scoping
Each node carries one or more cluster roles set in the HOCON `roles` list. Site nodes carry both a base `"Site"` role and a site-specific role (`"site-{SiteId}"`, e.g. `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster, not on any other site's nodes. Central singletons use no role scope — all central nodes share the `"Central"` role.
Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role.
### Dual-node recovery
Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. No startup ordering dependency exists, and no manual intervention is required. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.
Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.
### Cluster singletons hosted
The Host wires the following singletons. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern; each singleton's behaviour is documented in the owning component.
The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component.
**Central singletons (active central node, no role scope):**
**Central singletons (oldest `Up` central node, no role scope):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
| `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) |
| `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) |
| `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) |
| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) |
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) |
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History |
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) |
**Site singletons (active site node, scoped to `"site-{SiteId}"` role):**
**Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):**
| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
@@ -173,29 +202,29 @@ Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator
services.AddClusterInfrastructure();
```
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section (wrong strategy, `MinNrOfMembers != 1`, `DownIfAlone = false`, fewer than two seed nodes) throws an `OptionsValidationException` at startup rather than booting into a broken cluster.
This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there).
### Checking active-node status
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root):
Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership:
```csharp
// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode
{
get
{
var system = _akkaService.ActorSystem;
if (system == null) return false;
if (system == null)
return false;
var cluster = Cluster.Get(system);
var self = cluster.SelfMember;
if (self.Status != MemberStatus.Up) return false;
var leader = cluster.State.Leader;
return leader != null && leader == self.Address;
return ClusterActivityEvaluator.SelfIsOldest(cluster);
}
}
```
This returns `false` while the actor system is warming up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes.
This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.
## Configuration
@@ -205,13 +234,14 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries; both nodes list both themselves and their partner. |
| `SplitBrainResolverStrategy` | `string` | `"keep-oldest"` | Must be `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Cluster must be stable for this duration before the resolver acts to down unreachable nodes. |
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. |
| `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. |
| `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. |
| `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. |
| `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. |
| `DownIfAlone` | `bool` | `true` | Must be `true`. See split-brain resolution above. |
| `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. |
| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. |
### `ScadaBridge:Node`
@@ -241,7 +271,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "keep-oldest",
"SplitBrainResolverStrategy": "auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -251,7 +281,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
}
```
`DownIfAlone` is not present in the docker files because its default value of `true` is correct and `ClusterOptionsValidator` rejects `false`.
`DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert.
## Dependencies & Interactions
@@ -261,22 +291,26 @@ This returns `false` while the actor system is warming up — the safe-by-defaul
- [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
- [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the cluster leader.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member.
- Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md).
## Troubleshooting
### Node fails to join cluster on startup
`ClusterOptionsValidator` rejects fewer than two seed nodes, a non-`keep-oldest` strategy, `MinNrOfMembers != 1`, or `DownIfAlone = false` at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.
A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).
### Singleton not starting after failover
If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this.
### Two live clusters (split-brain)
### Two live clusters (dual-active)
If `DownIfAlone = false` were accepted (the validator rejects it), the oldest node could run alone while the younger forms its own cluster, producing two live clusters with divergent singleton state and dual MS SQL writers on central. `ClusterOptionsValidator` makes this configuration impossible to boot.
Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby.
Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage.
### Graceful shutdown takes longer than expected
@@ -285,6 +319,7 @@ If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShut
## Related Documentation
- [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md)
- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md)
- [Host](./Host.md)
- [Site Runtime](./SiteRuntime.md)
- [Health Monitoring](./HealthMonitoring.md)
docs/components/Communication.md
+70 -31
View File
@@ -1,6 +1,6 @@
# Central–Site Communication
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides two independent transports — Akka.NET `ClusterClient` for command/control and gRPC server-streaming for real-time data — wired together through a pair of actors that each cluster registers with the `ClusterClientReceptionist`.
The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports, **all now gRPC-based across the boundary** (Akka `ClusterClient` was removed in Phase 4 of the ClusterClient→gRPC migration, 2026-07-23): gRPC command/control in both directions (`CentralControlService` on central via `GrpcCentralTransport`; `SiteCommandService` on the site via `GrpcSiteTransport`), gRPC server-streaming for real-time data (`SiteStreamService`), and plain token-gated HTTP for the deployment-config fetch. Cross-cluster Akka remoting and `ClusterClientReceptionist` are gone; Akka remoting is now intra-cluster only.
## Overview
@@ -18,27 +18,52 @@ DI registration is called from the Host composition root via `AddCommunication`.
## Key Concepts
### Two transports, two concerns
### Three transports, three concerns
| Transport | Direction | Purpose |
|-----------|-----------|---------|
| Akka.NET `ClusterClient` | bidirectional (command/control) | Deployments, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications |
| gRPC server-streaming (`SiteStreamService`) | site → central | Real-time attribute value and alarm state changes |
| Transport | Who dials | Data direction | Purpose |
|-----------|-----------|----------------|---------|
| gRPC command/control (`CentralControlService`) | **site dials central** | site → central | Heartbeats, health reports, notification submit/status, audit + cached-telemetry ingest, reconcile |
| gRPC command/control (`SiteCommandService`) | **central dials the site** | central → site | Deploy notifies, lifecycle, OPC UA, remote queries, subscribe/unsubscribe handshake, snapshots, parked, route, failover |
| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs |
| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` |
The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa.
The transports are independent. A gRPC stream interruption does not affect in-flight command/control calls, and vice versa.
**`SiteStreamService`'s gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the `SiteStreamService` server** and **central is the client**. Command/control is different: it runs on its own two services — `CentralControlService` (central-hosted, site dials in) and `SiteCommandService` (site-hosted, central dials in). So a central node now DOES host a gRPC server (`CentralControlService`), unlike before Phase 4. The two legacy `Ingest*` unary RPCs on the site-hosted `SiteStreamService` remain dead in the shipped topology (no site dials a site for ingest); audit telemetry is pushed site → central over `CentralControlService`, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site's `SiteStreamService`.
**No transport carries transport encryption; two of the three now carry authentication.**
- **Akka remoting — unauthenticated, but intra-cluster only.** `BuildHocon` emits no `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting is plaintext and open to anything that can reach the remoting port — but as of Phase 4 remoting no longer crosses the site↔central boundary, so this exposure is pair-internal.
- **gRPC — authenticated by preshared key since 2026-07-22.** The listeners are still **h2c** — `ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` — but `ControlPlaneAuthInterceptor` gates the command/control and streaming services alike: every method under `/sitestream.SiteStreamService/` (including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows) **and** the Phase-4 `CentralControlService` (site→central) / `SiteCommandService` (central→site). It is fail-closed (no key ⇒ everything refused, and `StartupValidator` will not boot a site node in that state), compares with `CryptographicOperations.FixedTimeEquals`, and rejects with `PermissionDenied`. The caller attaches the key via `ControlPlaneCredentials`, which binds `CallCredentials` to each channel so unary and streaming calls are covered uniformly. Keys are **per site** (`SB-GRPC-PSK-<siteId>`), so a compromised site yields only its own. `LocalDbSyncAuthInterceptor` shares the listener and keeps its own separate key on `/localdb_sync.v1.LocalDbSync/` — the two authenticate different peers (central vs. the pair partner) and are never shared.
- **HTTP config fetch — token-authenticated.** Its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`).
A bearer PSK over plaintext h2c is readable and replayable by anyone on the path, so the design still assumes a trusted network between central and sites — but the bar is now "read the traffic" rather than "reach the port". TLS on these listeners is the follow-on hardening and would not change the key design. Operational detail: [`docs/deployment/topology-guide.md`](../deployment/topology-guide.md).
### Notify-and-fetch: the deployment-config HTTP path
An instance deployment does not carry its flattened configuration inside the command message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`), carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP:
```csharp
// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs
var url = $"{centralFetchBaseUrl.TrimEnd('/')}/api/internal/deployments/{Uri.EscapeDataString(deploymentId)}/config";
using var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("X-Deployment-Token", token);
```
`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This path was originally introduced because a flattened config could exceed the default 128 KB Akka frame size when command/control rode ClusterClient — the oversized message was silently dropped and the deploy hung to its timeout (see `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). That Akka frame limit no longer applies now that command/control rides gRPC (`SiteCommandService`, 4 MB cap), but notify-and-fetch remains the deploy path. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline — now over gRPC, so it is no longer at risk of the 128 KB drop.
### Hub-and-spoke topology
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes.
Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `GrpcSiteTransport` channel pair per site (dialling `SiteCommandService`); each site maintains a single `GrpcCentralTransport` channel pair pointed at both central nodes (dialling `CentralControlService`).
### `SiteEnvelope` routing
Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site:
```csharp
// CommunicationService.cs — deployment pattern
public async Task<DeploymentStatusResponse> DeployInstanceAsync(
string siteId, DeployInstanceCommand command, CancellationToken cancellationToken = default)
// CommunicationService.cs — deployment pattern (notify-and-fetch)
public async Task<DeploymentStatusResponse> RefreshDeploymentAsync(
string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<DeploymentStatusResponse>(
@@ -70,12 +95,14 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
## Architecture
> **Phase 4 note (2026-07-23).** The `ClusterClient` mechanics described in this section — `SiteEnvelope` routing via `ClusterClient.Send("/user/site-communication", …)`, one `ClusterClient` per site on `CentralCommunicationActor`, the site's outbound `_centralClient`, and `ClusterClientReceptionist` registration — were **replaced by gRPC command/control** in Phase 4 of the ClusterClient→gRPC migration. Central→site commands now go over `GrpcSiteTransport` → the site-hosted `SiteCommandService` (per-site NodeA→NodeB failover channel pair), and site→central messages over `GrpcCentralTransport` → the central-hosted `CentralControlService` (sticky central-a→central-b pair). Endpoints are dialled directly (no receptionist, no cross-cluster actor discovery). The per-site address load below still runs, but it now feeds a **per-site gRPC-endpoint cache** (from `GrpcNodeAAddress`/`GrpcNodeBAddress`) consumed by a `SitePairChannelProvider`, not a `ClusterClient` contact set. See [Component-Communication.md](../requirements/Component-Communication.md) for the current design.
### Central-side: `CentralCommunicationActor`
`CentralCommunicationActor` is a `ReceiveActor` created at `/user/central-communication` and registered with `ClusterClientReceptionist` so the site's `ClusterClient` can locate it. It owns:
- A `Dictionary<string, (IActorRef Client, ImmutableHashSet<string> ContactAddresses)>` keyed by site identifier — one `ClusterClient` per site.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database, parses `NodeAAddress` and `NodeBAddress` into Akka receptionist paths (`{addr}/system/receptionist`), and pipes a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` creates, updates, or stops `ClusterClient` actors based on the diff.
- A `RefreshSiteAddresses` periodic timer (60-second cadence, starting immediately). Each tick fires `LoadSiteAddressesFromDb`, which reads every `Site` row from the database and (as of Phase 4) builds a **per-site gRPC-endpoint cache** from `GrpcNodeAAddress`/`GrpcNodeBAddress`, piping a `SiteAddressCacheLoaded` message back to Self. `HandleSiteAddressCacheLoaded` reconciles the diff, updating the `SitePairChannelProvider` that `GrpcSiteTransport` uses to dial each site's `SiteCommandService`. (Before Phase 4 this parsed the Akka `NodeAAddress`/`NodeBAddress` into receptionist paths and created/updated/stopped a `ClusterClient` per site.)
- Proxy references to `NotificationOutboxActor` and `AuditLogIngestActor` cluster singletons, injected post-construction via `RegisterNotificationOutbox` / `RegisterAuditIngest` messages from the Host. Messages that arrive before the proxy is registered are answered with a non-accepted ack (notifications) or an empty reply (audit), so the site retries without data loss.
- Fanout of `SiteHealthReport` to the peer central node via `DistributedPubSub`, keyed on the `site-health-replica` topic, so both central nodes' aggregators stay in sync regardless of which central node the site's `ClusterClient` load-balanced the report to.
@@ -86,7 +113,7 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen
`SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns:
- An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`.
- A `Timers`-based heartbeat (default 5-second interval, first tick after 1 second). Each tick sends a `HeartbeatMessage` with `IsActive` stamped from the Akka `Cluster` leader check — the node is active when its `MemberStatus` is `Up` and it holds cluster leadership.
- A `Timers`-based heartbeat on `CommunicationOptions.ApplicationHeartbeatInterval` (default 5 s; deliberately distinct from the Akka.Remote `TransportHeartbeatInterval`, so retuning the transport failure detector cannot silently retune the health heartbeat). Each tick sends a `HeartbeatMessage` whose `IsActive` is stamped from `ActiveNodeEvaluator.SelfIsOldestUp` — the node is active when it is the **oldest `Up` member**, *not* when it holds cluster leadership (`SiteCommunicationActor.cs:517-518`). A throwing active-check is caught and reported as `IsActive = false`.
- Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall.
Site-to-central messages (health reports, audit batches, notification submissions) are sent via:
@@ -108,9 +135,9 @@ A malformed address for one site does not abort the refresh loop — the actor c
### gRPC real-time data transport
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, a gRPC server-streaming service defined in `sitestream.proto`.
Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, defined in `sitestream.proto`. The **server runs on every site node and the client runs on central** — central dials in to receive the stream (see the transport table above).
**Site-side** — `SiteStreamGrpcServer` (Kestrel HTTP/2, port 8083):
**Site-side** — `SiteStreamGrpcServer` (Kestrel h2c, HTTP/2 only, port 8083):
- Implements `SiteStreamService.SiteStreamServiceBase`.
- For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly).
@@ -144,7 +171,7 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
**Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`:
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per site identifier. On `GetOrCreate`, it compares the cached client's `Endpoint` to the requested endpoint and atomically replaces a stale client (different endpoint — NodeA→NodeB failover flip, or an edited address) with a fresh one.
- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per **`(site, endpoint)` pair** — a `ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient>`. The key was widened from site-only to fix an arch-review High: with a site-only key, one debug session's NodeA→NodeB failover flip disposed a channel another session was still using. `GetOrCreate` therefore no longer disposes on endpoint mismatch; both of a site's node channels coexist, and site *removal* (`RemoveSiteAsync`) is the only shared-disposal path. The trade-off is that an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site.
- `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread.
### Debug stream session lifecycle
@@ -162,18 +189,22 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg)
### Proto definition summary
```proto
// Protos/sitestream.proto
// Protos/sitestream.proto — six RPCs, all served by the SITE
service SiteStreamService {
rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent);
rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent);
rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck);
rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse);
rpc PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse);
}
```
`SubscribeInstance` carries the real-time data stream. The other three RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`) serve the Audit Log component's gRPC telemetry push and reconciliation pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there.
Two are server-streaming: `SubscribeInstance` carries the per-instance real-time stream; `SubscribeSite` is the **site-wide, alarm-only** stream (no instance filter, attribute updates never carried) that feeds the aggregated central live alarm cache. The four unary RPCs serve the Audit Log and Site Call Audit push/pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. As noted above, the two `Ingest*` RPCs are dead in the shipped topology (no central gRPC server exists for a site to dial); the two `Pull*` RPCs are live, with central as the caller.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–21) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work.
`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–23) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. Field numbers are never reused and evolution is additive only.
The generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out, so editing `sitestream.proto` does not regenerate on build — regeneration is a manual toggle-build-copy-untoggle.
## Usage
@@ -181,10 +212,10 @@ Central callers interact through `CommunicationService`, which wraps each comman
| Pattern | Method | Timeout |
|---------|--------|---------|
| Instance deployment | `DeployInstanceAsync` | 120 s |
| Instance deployment (notify-and-fetch) | `RefreshDeploymentAsync` | 120 s |
| Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s |
| Artifact deployment | `DeployArtifactsAsync` | 60 s |
| Integration routing | `RouteIntegrationCallAsync` | 30 s |
| Integration routing (Inbound API routed-site-script) | `RouteToCallAsync`, `RouteToGetAttributesAsync`, `RouteToSetAttributesAsync`, `RouteToWaitForAttributeAsync` | 30 s (`IntegrationTimeout`) |
| Debug snapshot | `RequestDebugSnapshotAsync` | 30 s |
| Remote queries | `QueryEventLogsAsync`, `QueryParkedMessagesAsync`, etc. | 30 s |
| OPC UA tag browse | `BrowseNodeAsync` | 30 s |
@@ -197,24 +228,28 @@ For real-time streaming, callers use `DebugStreamService.StartStreamAsync`, whic
## Configuration
All options are bound from the `Communication` section via `CommunicationOptions`:
All options are bound from the `ScadaBridge:Communication` section via `CommunicationOptions`:
| Key | Default | Description |
|-----|---------|-------------|
| `DeploymentTimeout` | `00:02:00` | Ask timeout for instance deployment commands. |
| `DeploymentTimeout` | `00:02:00` | Ask timeout for the `RefreshDeploymentCommand` round-trip (covers the site's HTTP config fetch and apply). |
| `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). |
| `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. |
| `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. |
| `IntegrationTimeout` | `00:00:30` | Ask timeout for integration routing and Inbound API routing. |
| `DebugViewTimeout` | `00:00:10` | Ask timeout for debug subscribe/unsubscribe handshake. |
| `NotificationForwardTimeout` | `00:00:30` | Ask timeout for notification submission forwarding. |
| `CentralContactPoints` | `[]` | Site-side: Akka addresses of central nodes, e.g. `akka.tcp://scadabridge@central-a:8081`. |
| `CentralGrpcEndpoints` | `[]` | Site-side: gRPC h2c endpoints of the central nodes' `CentralControlService`, e.g. `http://scadabridge-central-a:8083` (central's `CentralGrpcPort`, default 8083 — direct, not via Traefik, which is HTTP/1 only). **Required on Site nodes** (at least one); empty on Central nodes, which host `CentralControlService` and do not dial it. Replaces the former `CentralContactPoints` Akka-address list. |
| `GrpcKeepAlivePingDelay` | `00:00:15` | HTTP/2 keepalive PING interval on `SiteStreamGrpcClient`. |
| `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. |
| `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. |
| `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. |
| `TransportHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` heartbeat cadence. |
| `ApplicationHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` site→central heartbeat cadence. |
| `TransportHeartbeatInterval` | `00:00:05` | Akka.Remote transport failure-detector heartbeat interval (emitted into the HOCON by the Host). Distinct from the application heartbeat above. |
| `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. |
| `CentralFetchBaseUrl` | `""` | Base URL (Traefik/LB) the site uses to fetch deploy configs from central. Carried in `RefreshDeploymentCommand` so sites need no standing config; **empty makes a deploy impossible** — `DeploymentService` fails fast. |
| `PendingDeploymentTtl` | `00:05:00` | How long a staged `PendingDeployment` row and its fetch token stay valid. Must comfortably cover both site nodes' fetches within one deploy window. |
| `PendingDeploymentPurgeInterval` | `01:00:00` | Cadence of the central `pending-deployment-purge` singleton that sweeps TTL-expired staging rows. Hygiene only — the fetch endpoint already enforces the TTL. |
Three layers of dead-client detection protect the gRPC stream path:
@@ -227,27 +262,27 @@ Three layers of dead-client detection protect the gRPC stream path:
## Dependencies & Interactions
- [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the active/standby leader model that `SiteCommunicationActor`'s `IsActive` check and `CentralCommunicationActor`'s `DistributedPubSub` fanout both depend on.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides ClusterSingleton and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name for intra-cluster remoting. (`ClusterClientReceptionist` is no longer used — cross-cluster messaging is gRPC as of Phase 4.) `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deployments, lifecycle commands, and artifact deployments to sites.
- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service.
- [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly.
- [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts `IngestAuditEvents`, `IngestCachedTelemetry`, and `PullAuditEvents` RPCs. `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` ClusterClient messages to the `AuditLogIngestActor` proxy.
- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. The `Ingest*` pair on the site-hosted service stays unused in the shipped topology; sites push audit telemetry site→central over gRPC to `CentralControlService`, which routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site's `SiteStreamService`.
- [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls.
- [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`.
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component.
- [Management Service (#18)](./ManagementService.md) — `ManagementActor` runs at `/user/management` on central and is reached **in-process** through `ManagementActorHolder`; the CLI connects over HTTP, not any cluster transport. (It was `ClusterClientReceptionist`-registered until 2026-07-22, for a CLI that was never built that way.) After Phase 4 this component uses no `ClusterClient` at all — its cross-cluster connections are the gRPC `CentralControlService` / `SiteCommandService` transports. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the HTTP deploy-config transport, mapped in the central-role block alongside `/api/audit/*` and `/management`.
- Design spec: [Component-Communication.md](../requirements/Component-Communication.md).
## Troubleshooting
### A site's commands fail immediately
Check that `NodeAAddress` and `NodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `ClusterClient` is created and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
Check that `GrpcNodeAAddress` and `GrpcNodeBAddress` are populated in the site configuration — if both are empty, `CentralCommunicationActor` logs a warning and skips that site on every refresh, so no `SiteCommandService` channel is built and all commands timeout. `CommunicationService.RefreshSiteAddresses()` triggers an on-demand refresh after an address is added.
### Commands are timing out but the site is reachable
A single malformed address string for one site can silently prevent `ClusterClient` creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The actor parse-guard catches the `ActorPath.Parse` exception per-site so the rest of the refresh proceeds.
A single malformed gRPC endpoint string for one site can silently prevent channel creation for that site while other sites are unaffected. Check the logs for a `Warning` line from `HandleSiteAddressCacheLoaded` naming the offending site. The per-site parse-guard skips the bad entry so the rest of the refresh proceeds.
A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means `LoadSiteAddressesFromDb` itself threw (typically a SQL connection error); the cache is left stale until the next successful refresh.
@@ -257,6 +292,10 @@ A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means
After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session.
### Deployments fail immediately with a config-fetch error
The site received the `RefreshDeploymentCommand` over gRPC command/control (`SiteCommandService`) but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired".
### Heartbeats arrive but health reports do not
`SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`.
docs/components/DeploymentManager.md
+20 -6
View File
@@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers:
- **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call.
- **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment.
The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running.
Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip.
### Notify-and-fetch: the config does not travel in the Akka message
A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path.
This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit.
Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour).
The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change.
### Per-instance operation lock
`OperationLockManager` holds a `Dictionary<string, LockEntry>` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel.
@@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment
3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written.
4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent.
5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI.
6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`.
6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before.
7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails.
8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`.
9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user.
@@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 6–7) writes `DeploymentStatus.Fail
```csharp
// DeploymentService.DeployInstanceAsync — exception handler
var isTimeout = ex is TimeoutException or OperationCanceledException;
var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException;
record.Status = DeploymentStatus.Failed;
record.ErrorMessage = isTimeout
@@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De
- [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg.
- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface.
- [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active.
- [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests.
- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s.
- [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`.
## Troubleshooting
@@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent
The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one.
### A deployment fails with a config-fetch error
The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout.
### DeleteOrphaned audit entry
The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.
docs/components/Host.md
-1
View File
@@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
`SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates:
- `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points.
- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby.
- `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`.
- `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors.
docs/components/ManagementService.md
+7 -4
View File
@@ -56,17 +56,18 @@ Mutating handlers that call repositories directly invoke `AuditAsync` (backed by
### Actor lifecycle and registration
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and registers it with `ClusterClientReceptionist`:
`AkkaHostedService` (in the Host) creates the `ManagementActor` under the path `/user/management` and publishes it to `ManagementActorHolder`, which is the only way anything reaches it:
```csharp
var mgmtActor = _actorSystem!.ActorOf(
Props.Create(() => new ManagementActor(_serviceProvider, mgmtLogger)),
"management");
ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor);
var mgmtHolder = _serviceProvider.GetRequiredService<ManagementActorHolder>();
mgmtHolder.ActorRef = mgmtActor;
```
A `ClusterClientReceptionist.Get(_actorSystem).RegisterService(mgmtActor)` call sat between those two statements until 2026-07-22. It was deleted because nothing ever sent to it: the CLI it was built for uses HTTP, not ClusterClient.
`ClusterClientReceptionist` advertises the actor to `ClusterClient` senders without requiring them to join the Akka cluster. The `ManagementActorHolder.ActorRef` property is then the bridge from the HTTP endpoint (which runs in ASP.NET Core middleware) into the Akka actor world.
The actor declares an explicit supervisor strategy — one-for-one with Resume and no retry limit — to match the coordinator-actor convention and remain correct if child actors are added later.
@@ -154,9 +155,11 @@ Content-Type: application/json
A successful response is HTTP 200 with the JSON result. An authorization failure is HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`.
### Sending a command via ClusterClient
### Sending a command in-process
The `ManagementActor` is also reachable from any `ClusterClient` that has a contact point into the central cluster. The actor is registered under `/system/receptionist` with the path `/user/management`. Callers construct and `Tell` a `ManagementEnvelope` and expect one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply.
`ManagementEnvelope` is also the in-process contract: a caller holding `ManagementActorHolder.ActorRef` asks the actor directly and expects one of `ManagementSuccess`, `ManagementError`, or `ManagementUnauthorized` in reply. `ManagementEndpoints` is that caller.
There is **no** out-of-process actor path. The actor was advertised via `ClusterClientReceptionist` until 2026-07-22, so a `ClusterClient` with a contact point into the central cluster could `Tell` it a `ManagementEnvelope`; no caller ever did, and the registration is gone. The HTTP endpoints above are the only remote management surface.
## Command Groups
docs/components/SiteRuntime.md
+3 -3
View File
@@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`.
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
@@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
For redeployment (instance already running), the existing actor is stopped and watched:
@@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
### Standby replication
`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation.
Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
## Usage
docs/components/StoreAndForward.md
+4 -3
View File
@@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta
## Overview
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby.
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted.
The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`:
- `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook.
- `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`.
- `ReplicationService` — fire-and-forget buffer replication to the standby.
- `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`.
- `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack.
- `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section.
@@ -129,7 +128,9 @@ else
### Async replication to standby
`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path.
Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`.
The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound.
The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover).
docs/components/TreeView.md
+23 -1
View File
@@ -41,9 +41,31 @@ In `Single` mode the component uses `SelectedKey` / `SelectedKeyChanged` (two-wa
When `ContextMenu` is non-null, right-clicking any row suppresses the browser default and positions a Bootstrap `dropdown-menu show` div at the cursor coordinates using `position: fixed`. An invisible overlay behind the menu dismisses it on click-outside; Escape also dismisses it. The menu receives the `TItem` of the right-clicked node, so the consumer's fragment can branch on node type.
### Keyboard navigation & accessibility (WAI-ARIA tree pattern)
Delivered 2026-08-01 (M10 residual R7). The component implements the full [WAI-ARIA tree pattern](https://www.w3.org/WAI/ARIA/apg/patterns/treeview/).
**Roving tabindex.** Exactly one `li[role="treeitem"]` carries `tabindex="0"` at a time; every other node is `tabindex="-1"`, so the whole tree is a single Tab stop. The target is resolved once per render by `ResolveTabbableKey()`: the node the user last landed on (`_focusedKey`) while it is still visible → else the currently `SelectedKey` node → else the first visible node. A keyboard move sets `_focusNeedsApply`, and `OnAfterRenderAsync` pulls browser focus onto the new node via a per-node `ElementReference`, guarded by the same `JSException` / `JSDisconnectedException` / `InvalidOperationException` triple the context-menu focus uses (under bUnit there is no real focus, so it is a safe no-op).
| Key | Behaviour |
| --- | --- |
| `ArrowDown` / `ArrowUp` | Move to the next / previous **visible** node (a collapsed branch's children are skipped). No-op at the ends. |
| `ArrowRight` | Collapsed branch → expand (focus stays); expanded branch → move to first child; leaf → no-op. |
| `ArrowLeft` | Expanded branch → collapse (focus stays); otherwise → move to parent (via `BuildParentLookup()`). No-op at a root leaf. |
| `Home` / `End` | Move to the first / last visible node. |
| `Enter` / `Space` | Activate the node — routed to the *same* path a mouse click takes (`OnContentClick` in `Single` mode, `OnCheckboxToggle` in `Checkbox` mode), so selection semantics never diverge between input modes. |
The visible order is produced by `BuildVisibleNodes()`, which mirrors `RenderNode`'s visibility rules exactly — **the two must stay in step**, or arrow navigation will skip or invent rows.
**ARIA attributes.** `ul[role=tree]` root, `ul[role=group]` for children, `li[role=treeitem]` nodes carrying `aria-level` (1-based), `aria-posinset`, `aria-setsize`, `aria-expanded` (branches only), and `aria-selected` (selectable / checkbox trees only — absent on a non-selectable tree).
**Event scoping.** `@onkeydown:stopPropagation` sits on the `<li>` (so a nested node's keypress is not also handled by its ancestors) and on the chevron, the checkbox, and the `.tv-content` slot — so a consumer's own buttons and inputs inside `NodeContent` keep their key handling. Browser scroll-on-Space/Arrow is suppressed by a small **native** inline `onkeydown` on the root `<ul>` that calls `preventDefault()` only for the navigation keys and only when the event target is the treeitem itself; Blazor's `preventDefault` directive is all-or-nothing per element, so putting it on the `<li>` would both trap Tab inside the tree and cancel Enter/Space activation of consumer buttons. (The app sets no Content-Security-Policy, so the inline handler executes.)
Covered by `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Shared/TreeViewKeyboardNavigationTests.cs` (31 bUnit tests).
## Architecture
The component is a single `@typeparam` `.razor` file with a private `void RenderNode(TItem item, int depth)` local function that recurses the tree at render time — no intermediate view model is built inside the component. Every `<li>` carries `@key="key"` so Blazor can diff the list efficiently.
The component is a single `@typeparam` `.razor` file with a private `void RenderNode(TItem item, int depth, int posInSet, int setSize)` local function that recurses the tree at render time — no intermediate view model is built inside the component. Every `<li>` carries `@key="key"` so Blazor can diff the list efficiently.
`IJSRuntime` is injected for two purposes: reading/writing `sessionStorage` for expansion persistence, and setting `input.indeterminate` for tri-state checkboxes. Both call sites guard `JSDisconnectedException` so a disconnected circuit never throws out of the lifecycle methods.
docs/deployment/installation-guide.md
+8
View File
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
## Upgrading a Site Pair
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
`TombstoneRetention` bound on how long one node may stay offline.
## Post-Installation Verification
1. Start the service: `sc.exe start ScadaBridge-Central`
docs/deployment/production-checklist.md
+4 -1
View File
@@ -44,6 +44,8 @@
- [ ] Windows Service account has minimum necessary permissions
- [ ] Log directory permissions restrict access to service account and administrators
- [ ] SMTP credentials use OAuth2 Client Credentials (preferred) or secure Basic Auth
- [ ] EWS transport (`Transport=Ews`): endpoint is an absolute `https://` URL and auth mode is Basic — Basic requires TLS and this is enforced at the write gate, the delivery adapter, and the sender
- [ ] EWS transport: the Exchange service-account password is rotated on the account-owner's schedule, and the SMTP configuration row is updated in the same change
- [ ] API keys for Inbound API are generated with sufficient entropy (32+ chars)
### Network
@@ -51,7 +53,8 @@
- [ ] DNS resolution works between all cluster nodes
- [ ] Firewall rules permit Akka.NET remoting (TCP 8081)
- [ ] Firewall rules permit LDAP (TCP 636 for LDAPS)
- [ ] Firewall rules permit SMTP (TCP 587 for TLS)
- [ ] Firewall rules permit SMTP (TCP 587 for TLS) — SMTP transport only
- [ ] EWS transport (`Transport=Ews`): firewall rules permit outbound HTTPS (TCP 443) from central nodes to the Exchange CAS instead of SMTP 587
- [ ] Firewall rules permit SQL Server (TCP 1433) from central nodes only
- [ ] Load balancer health check configured against `/health/ready`
docs/deployment/topology-guide.md
+76 -8
View File
@@ -88,17 +88,19 @@ Both central nodes must be configured as seed nodes for each other:
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@central-01.example.com:8081",
"akka.tcp://scadabridge@central-02.example.com:8081"
"akka.tcp://scadabridge@central-02.example.com:8081",
"akka.tcp://scadabridge@central-01.example.com:8081"
]
}
}
}
```
> **Seed order is load-bearing — each node lists ITSELF first** (decision 2026-07-22). Note Node B's list is the reverse of Node A's. Akka only lets `seed-nodes[0]` form a *new* cluster, so a node listing its partner first can never boot while that partner is down. `StartupValidator` rejects the boot if the ordering is wrong, comparing host **and** port; use the same spelling of the hostname in `NodeHostname` and in the seed URI, since Akka does no DNS canonicalisation (`central-02` and `central-02.example.com` are different seed identities). See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
### Cluster Behavior
- **Split-brain resolver**: Keep-oldest with `down-if-alone = on`, 15-second stable-after.
- **Split-brain resolver**: `auto-down` (`AutoDowning` provider, `auto-down-unreachable-after` = 15s) since the 2026-07-21 availability-over-partition-safety decision — the leader among the *reachable* members downs the unreachable peer, so a hard crash of **either** node fails over. Accepted trade: a real partition leaves both sides active until an operator restarts one. `keep-oldest` (with `down-if-alone = on`) remains a supported `SplitBrainResolverStrategy` value, but in a two-node cluster it cannot survive a crash of the oldest node. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
- **Minimum members**: `min-nr-of-members = 1` — a single node can form a cluster.
- **Failure detection**: 2-second heartbeat interval, 10-second threshold.
- **Total failover time**: ~25 seconds from node failure to singleton migration.
@@ -145,18 +147,84 @@ Each site has its own two-node cluster:
}
```
> **Site Node B reverses this list** — `site-01-b` first, `site-01-a` second — per the self-first seed rule above. It applies to site pairs exactly as it does to the central pair: without it, `site-01-b` cannot boot while `site-01-a` is down.
### Site Cluster Behavior
- Same split-brain resolver as central (keep-oldest).
- Same split-brain resolver as central (`auto-down`, per the 2026-07-21 decision — see the Central Cluster Behavior note above).
- Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication).
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
share a SQLite file.
### Site Pair Upgrades — stop and start BOTH nodes together
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
nothing left to suppress the stale row the returning node still holds. Within the retention window
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.
### Central-Site Communication
- Sites connect to central via Akka.NET remoting.
- The `Communication:CentralSeedNode` setting in the site config points to one of the central nodes.
- If that central node is down, the site's communication actor will retry until it connects to the active central node.
Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed
in Phase 4 of the ClusterClient→gRPC migration (2026-07-23), and Akka remoting no longer crosses the
boundary at all:**
- **gRPC command/control** — both directions, on sticky-failover channel pairs, dialled directly (no
receptionist, no "active central" to identify — each side dials both of the peer's node endpoints):
- *Site → central* to the central-hosted **`CentralControlService`** (`GrpcCentralTransport`): the
site lists the central nodes' gRPC endpoints in `ScadaBridge:Communication:CentralGrpcEndpoints`
(e.g. `http://scadabridge-central-a:8083`, the central's `CentralGrpcPort`, default 8083 — direct
h2c, **not** via Traefik, which is HTTP/1 only). A Site node must list at least one; central nodes
leave it empty.
- *Central → site* to the site-hosted **`SiteCommandService`** (`GrpcSiteTransport`): central dials
the site's `GrpcNodeAAddress` / `GrpcNodeBAddress` (from the Site entity), NodeA→NodeB failover.
- **gRPC streaming + audit pull** — real-time data and audit/telemetry pull on the site-hosted
**`SiteStreamService`**. Note the direction is inverted from the data flow: each **site node hosts
the server** on `GrpcPort` (default 8083, h2c) and central dials in.
- **Plain HTTP** — the deploy config itself, fetched by the site with a per-deployment token.
#### gRPC control-plane preshared key (required)
Every site node must set `ScadaBridge:Communication:GrpcPsk`, and central must hold the same
value for that site. **`StartupValidator` refuses to boot a site node without it**, deliberately:
the gate is fail-closed, so an unset key would leave the node joined, healthy-looking and
answering heartbeats while refusing every gRPC call — no live subscriptions, no audit pull, no
cached-telemetry ingest.
| Side | Where the key lives |
|---|---|
| Site node (both nodes of the pair, identical) | `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}` |
| Central | secret `SB-GRPC-PSK-<siteId>` in its store — **or** `ScadaBridge:Communication:SitePsks:<siteId>` |
The store is the source that matters in production, because sites are added at runtime and their
keys cannot be enumerated in configuration at boot; `SitePsks` covers a host running without a
master key (the docker rig) and one-off pins.
One key **per site**, never one for the fleet: a compromised site must not yield another site's
key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
for database replication, a different trust relationship on the same listener.
**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
including in the on-host `deploy/` overlays.
The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the
path. That is the accepted posture today — the same trusted-network assumption the boundary
already made, now with authentication rather than none. TLS on these listeners is follow-on
hardening and needs no change to the key design.
## Scaling Guidelines
docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md
+40
View File
@@ -10,6 +10,46 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
more, so the 128 000-byte frame constraint does not apply to it in any form.
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
still fetches over HTTP, so that half of the original fix stands.
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped.
## Amendment (2026-07-23) — Akka frame-size class RETIRED for site↔central command/control
Phase 4 of the ClusterClient→gRPC migration deleted the Akka `ClusterClient` site↔central transport
entirely. **All site↔central command/control now rides gRPC** — central→site over `SiteCommandService`
(`GrpcSiteTransport`) and site→central over `CentralControlService` (`GrpcCentralTransport`), each with
the gRPC default 4 MB message cap and **no per-message Akka frame limit**. Consequently:
- The 128 KB Akka `maximum-frame-size` constraint **no longer applies to any site↔central command
message**, including `DeployArtifactsCommand`, which still carries its payload inline but now travels
over gRPC.
- The **silent frame-drop failure mode** described below — the transport dropping one oversized message
while heartbeats keep flowing and the deploy hangs to its timeout — **cannot occur on that path any
more.** An over-cap gRPC message is *rejected* with an error, not silently discarded.
- The notify-and-fetch deploy path (the 2026-06-26 resolution) still stands and remains the deploy
mechanism; it is simply no longer the *only* thing keeping large payloads off a frame-limited hop.
- Akka remoting is now intra-cluster only, so its frame size governs only pair-internal traffic.
See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md
+139
View File
@@ -0,0 +1,139 @@
# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone
**Date:** 2026-07-20 · **Status:** RESOLVED (2026-07-20) · **Severity:** High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood)
· **Area:** AuditLog / Site Telemetry
## Resolution (2026-07-20)
Fixed by letting an unresolvable row LEAVE the queue.
`SiteAuditTelemetryActor.OnCachedDrainAsync` now abandons the operational half of a cached row
whose tracking snapshot is still unresolvable after a grace period
(`SiteAuditTelemetryOptions.CachedTrackingGraceSeconds`, default **300 s**) and marks it
`Forwarded`. A row with no `CorrelationId` at all is abandoned immediately — it can never resolve.
**Marking `Forwarded` does not drop the audit data.** That state's role in this machine is "no
longer owed by the drain, still eligible for reconciliation", which is exactly the situation:
`ISiteAuditQueue.ReadPendingSinceAsync` covers `Forwarded` rows as well as `Pending` ones and
central dedups on `EventId`, so the reconciliation pull still delivers them. What is genuinely lost
is the operational (`SiteCalls`) half — unrecoverable regardless, since the tracking row it would
have been built from no longer exists.
Three further behaviours, each deliberate:
- **Inside the grace window the row is still retried.** A missing snapshot is normally a brief
write race (the audit row lands microseconds before the tracking row); abandoning on the first
failed lookup would discard the operational half of every cached call that lost that race.
- **A tracking-store THROW never abandons.** A throw is a store fault (locked, corrupt,
mid-restore), not a verdict about the row. Those rows stay `Pending` however old they are.
- **Logging is per drain pass, not per row.** One summary line each for abandoned, lookup-failed
(with the first exception attached) and deferred rows. The deferred case dropped to Debug — being
inside the grace window is normal operation, not a warning.
### Severity was revised UP during the fix
The original write-up called this a log flood with "no data loss", which understated it. The queue
is read **oldest-first with a fixed `BatchSize` (default 256)**, so once a batch's worth of
permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails
identically, and **never reaches the newer rows behind them**. The cached-telemetry path stalls
permanently. The audit half still reached central by reconciliation, so "no data loss" held — but
"Medium" did not.
**Fixed in:** `SiteAuditTelemetryActor.cs` (`AbandonUnresolvableAsync` + the rewritten skip
branches), `SiteAuditTelemetryOptions.cs` (`CachedTrackingGraceSeconds`).
**Tests:** `SiteAuditTelemetryActorTests` — `CachedDrain_OrphanRow_PastGrace_IsAbandoned_...`,
`..._InsideGrace_IsRetried_NotAbandoned`, `..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue`,
`..._TrackingStoreThrow_DoesNotAbandonTheRow`. The pre-existing
`CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash` was **retargeted, not deleted** —
it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while
the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two
abandonment tests go red; the two must-NOT-abandon guards stay green in both directions.
The diagnosis below is retained as the record of how it was found.
## Summary
`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup
returns `null` the row is **skipped and deliberately left Pending**
(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up".
Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the
same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of
unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.
Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart
and a container restart, until the audit database itself was discarded.
```
[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.
```
## Why the rows became unresolvable
Two independent stores must agree:
- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the
container at `/app/auditlog.db`**, not on the bind-mounted data volume);
- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the
consolidated `LocalDb:Path` database (bind-mounted).
Anything that resets one without the other strands every audit row that referenced it. The code
comment already anticipates the cause — *"possible if the audit row is older than the tracking
retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an
exotic one.
**Two realistic production triggers, neither requiring operator error:**
1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain
catches up — a long central outage, a large backlog — the snapshots are pruned out from under
still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's
LocalDb file from its peer while its container-local `auditlog.db` survives.
It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while
`auditlog.db` — being inside the container — survived.
## Why the current handling is not enough
Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is
now in a state it can never leave:
- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
- no terminal state, so it is retried for the life of the database;
- one Warning per row per pass, which buries every other log line on the node.
The "central reconciliation will pick it up" comment is about the **audit half** reaching central by
another path. That may well be true — but it does not release the row from the local drain queue,
which is what actually loops.
## Suggested fix
Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:
1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal
(`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count
rather than one per row per pass.
2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern
`MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`).
3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row
entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether
central accepts a packet with no tracking half.
(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.
## Reproduction
1. Run a site node until it has cached-call audit rows with tracking correlations.
2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`);
leave `auditlog.db` in place.
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.
## Notes
- **No data loss.** The audit rows are intact and still reach central by the reconciliation path;
what is broken is the local drain's ability to ever finish.
- Discovered while cleaning the rig after the LocalDb Phase 2 live gate
(`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related
"deleting an instance orphans its buffered messages" observation is recorded.
docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md
+475
View File
@@ -0,0 +1,475 @@
# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load
**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening
**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes
**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md)
**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code)
> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a
> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped.
> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved
> as written, with corrections annotated where its conclusions did not survive
> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).
---
## 0. ROOT CAUSE (verified)
**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and
resets the WAL out from under the container process, permanently poisoning that process's
connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved —
the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all.
Mechanism, step by step:
1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount
boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice
versa), so it believes it is the **only** connection.
2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to
0 bytes**. Because the read happened while the container was idle (a standby node, or a gap
between write bursts), nothing blocks the checkpoint. This is exactly the file signature
found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute.
3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping,
kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite
connection pool). That index says the WAL contains N frames; the file now has none. Every
subsequent statement — reads and writes both consult the WAL index — fails with
**`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10
`'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is
**permanent until the process reopens the database** (restart).
### Why the original brief's conclusions were wrong
- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by
the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56
mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only
because a standby issues ~zero LocalDb statements; the errors "followed the load" because
the load is what generates statements against an already-poisoned handle. Node-b's very
first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`)
failed — it had been poisoned for 3 minutes with nothing to say about it.
- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in
the same directory were healthy only because no host process ever read *them*. The minimal
repro poisons an arbitrary WAL database the same way.
- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby
was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.
### Verification (2026-07-20, all on the live rig + minimal repro)
1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened
site-a-a took the full soak load for **10+ minutes with zero errors** (the original model
predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
2. **One host read is sufficient and immediate:** a single
`sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"`
against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the
first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next
40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a
WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op,
`synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`:
- against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) +
`SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL);
- during an **idle window** (connections held open, WAL populated) → WAL reset
1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run
(200/200)** — the persistent variant, matching the rig.
### Consequences
1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is
the root cause, not a hygiene note.** One violation silently destroys the node's local
persistence until restart. This applies to *every* WAL SQLite file on the bind mount
(`scadabridge.db`, `store-and-forward.db` included), not just LocalDb.
2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the
copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g.
`docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the
macOS host against live files.
3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write
load indefinitely once nothing external touched its file.
4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):**
- **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`,
`CachedCallTelemetryForwarder`, `SiteEventLogger`) now log
`SqliteException` primary/extended codes (`sqlite 10/522`-style) via
`SqliteErrorCodes.Describe` / `DescribeSqliteError`.
- **DONE — §8 async-context bug** (see §8).
- **DONE — §9.5 load regression test** (see §9).
- **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for
persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change
(pool clear + master reopen + in-flight coordination) protecting against *external
interference only* — the trigger is operator/tooling action, now prevented at the
source, and on same-kernel production deployments external readers see the locks and
are safe. File as its own issue if production ever runs where a foreign-kernel reader
can touch the files.
---
> **Original brief follows, preserved as written on 2026-07-20 before root-causing.**
---
## 1. Summary
On a ScadaBridge site node, once the node is **active** and under sustained concurrent write
load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails
with:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
```
Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up
and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same
process, under the same load, are completely unaffected.
## 2. Why it matters
1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs
`[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping
events on the floor on the active node whenever the site is busy. `OperationTracking` writes
fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking
snapshot for …; skipping`).
2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same
database — including `native_alarm_state` (highest-volume table on the node) and
`sf_messages` — **and deletes the bespoke mechanisms that currently carry that data**
(`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting
over onto this store while removing the fallback would convert a logging defect into config
and buffer loss.
3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence,
**not sustained write load** — which is why this was not caught.
## 3. Exact symptom
Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was
active and under load:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.<ProcessWriteQueueAsync>b__0(SqliteConnection connection)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221
```
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137
at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148
```
Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at
`Open()`. Connections are being acquired successfully; the failure is on the write itself.
### Error-source distribution
Error-stack frames counted over one 3-minute window on the loaded node:
| Store | Backing file | Frames |
|---|---|---|
| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 |
| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 |
| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 |
| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 |
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
## 4. Evidence — what has been established
### 4.1 It tracks the load, not the node
The load was moved between the two site-a nodes by restarting the active one (the surviving node
becomes oldest-up and takes over):
| Node | Role | Under load | `disk I/O error` / 4 min |
|---|---|---|---|
| site-a-a | active | yes | 2 175 |
| site-a-a | standby (after restart) | no | **0** |
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0**
> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read
> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).
This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same
bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the
**same process**, are also **WAL-mode**, and are being written **concurrently under the same
load** — and they log zero errors. Only the LocalDb-managed file fails.
### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause**
> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an
> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at
> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.
Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted
database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is
excluded:
- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b,
**node-b** — whose files no host process had touched since a single baseline read, and which
had been error-free for the entire preceding period — began erroring immediately at a *higher*
rate.
- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load.
The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database
over a bind mount is still unsafe and should be avoided — it is just not the cause here.)
### 4.4 Not disk pressure
Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small:
`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0.
## 5. Reproduction
Fully reproducible in ~10 minutes on the local docker rig.
### 5.1 Rig prerequisites
Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:
- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`).
- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now
applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the
`/docker-entrypoint-initdb.d/` compose mounts are informational only — the official
`mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose
file). The manual workaround below is retained for historical context / older checkouts.
Original problem: after `infra/reseed.sh` dropped the volume, nothing created
`ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its
setup.sql poll. The by-hand equivalent:
```bash
cd ~/Desktop/ScadaBridge/infra
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f"
done
```
Then restart the app containers so EF migrations run, and restart central again after
`seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup).
### 5.2 Build the load generator
**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment
validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one.
**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice.
`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no
S&F rows, and will not reproduce this.
```bash
cd ~/Desktop/ScadaBridge
SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first
AUTH="--url http://localhost:9000 --username multi-role --password password"
# 1. Point the seeded external system at a refusing address (discard port).
$SB $AUTH external-system update --id 1 --name "Test REST API" \
--endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1"
# 2. Minimal template: no attributes, no compositions, no connection bindings.
$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id
$SB $AUTH --format json template script add --template-id <TID> --name "SoakCall" \
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
--code 'var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);'
# 3. Four instances on site-a (site id 1), then deploy each.
for i in 1 2 3 4; do
$SB $AUTH --format json instance create --name "soakgen-$i" --template-id <TID> --site-id 1
done
$SB $AUTH instance deploy --id <each instance id>
```
Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only
changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not
word-split, so pass them literally or use `${=AUTH}`.
### 5.3 Observe
```bash
# Identify the ACTIVE node — it is the one running the ScriptActors.
docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused"
docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused"
# Errors appear on that node within ~2 minutes of load starting.
docker logs --since 4m scadabridge-site-a-<active> 2>&1 | grep -c "disk I/O error"
```
Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a
sidecar in the container's network namespace:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \
-s localhost:8084/metrics | grep '^localdb_'
```
Do **not** query the databases with host-side `sqlite3` while containers are writing them.
## 6. Code map
### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs`
Facts relevant to the failure:
- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to
"anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is
not thread-safe.
- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per
operation, from many concurrent actors. Every call then runs
`PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF:
```csharp
conn.CreateFunction("zb_hlc_next", () => _clock.Next());
```
- The connection string is **only** `DataSource=<path>` (`:57`) — **connection pooling is left at
the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set.
- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`.
ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets
only `Path` and the replication block.
- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread
during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared
`HybridLogicalClock` from arbitrary threads.
### Failing call sites (ScadaBridge)
- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained
single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper.
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`),
`:260,266` (`GetStatusAsync`).
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`.
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8.
## 7. Hypotheses, ranked
> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant
> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side
> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The
> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522).
None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as
healthy WAL databases".
1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh
`SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every
acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn
than the legacy stores (which reuse a small number of connections), and is the clearest
structural difference between the failing and healthy databases. Suspect the interaction of
the pool with the long-lived `_master` connection and WAL index growth.
2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.**
Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared
WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load
with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than
the bind mount).** If the errors vanish, this is confirmed and the fix is environmental /
deployment-shaped rather than a library bug. **Run this test first — it is cheap and it
partitions the hypothesis space.**
3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF
callback during trigger execution can surface as a generic SQLite error at the statement
level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw
under contention (e.g. a spin/overflow path when many callers request stamps in the same
millisecond).
4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection
write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not
`SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding.
### The single highest-value next step
**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`),
which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would
likely settle this outright:
| Extended code | Meaning | Points at |
|---|---|---|
| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 |
| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem |
| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking |
Add the extended code to the exception logging (or attach a debugger / run the repro against a
local non-container build) before pursuing any fix.
## 8. Secondary defect in the same path — **FIXED 2026-07-20**
```
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException
```
`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a
real bug independent of the I/O errors, though it sits in the same write path and may be
contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context`
after an `await` inside an actor.
> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so
> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with
> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is
> worse than the logged one: depending on what the pool thread's thread-static cell slot holds,
> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes
> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on
> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed
> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and
> `Self` into fields at construction (both are thread-safe immutable handles) and use only those
> from the re-arm path. Regression test
> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing`
> forces the mock awaits to complete off the actor thread — which every pre-existing test
> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for
> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.
## 9. What a fix must satisfy
> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no
> host process touches the live files — verified 10+ min of soak load with zero errors, events
> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load
> test) would **not** have caught this — the trigger is an external reader, not load — but is
> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent
> writers × 250 inserts through fresh pooled connections against a registered/triggered table on
> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite
> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8.
1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the
active node.
2. No `Failed to record event` errors — site events are durably written under load.
3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters.
4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and
live gate still pass).
5. A regression test that would have caught this — i.e. a **concurrent-write load test** against
a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's
gate passed precisely because no test applied sustained concurrent write pressure.
## 10. Rig state as left
- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped
(`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`).
- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to
`http://scadabridge-restapi:5200` when done.
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed
and still generating load** on site-a.
- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names.
## 11a. Fix pass (2026-07-20, same day — all verified)
Everything actionable that this incident identified is now fixed (uncommitted on each repo's
current branch; ScadaBridge full solution builds clean, 0 warnings):
| # | Issue | Fix | Verification |
|---|---|---|---|
| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 |
| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite <primary>/<extended>` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) |
| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) |
| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 |
| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 |
Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0
consequence 4 for the rationale and the condition under which to file it).
## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC)
- Both site-a nodes restarted during verification, curing both poisonings. End state:
**site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on
both** under sustained load.
- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at
`http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and
generating load — the Phase 2 soak can now proceed on a clean baseline.
- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in
the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md
+68
View File
@@ -0,0 +1,68 @@
# Integration call routing (`IntegrationCallRequest`) is dead on both ends
**Date:** 2026-07-22 · **Status:** RESOLVED (DELETED 2026-07-23) · **Tracked:** Gitea
[#32](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/32) (filed 2026-07-23) · **Severity:**
Low (no runtime impact — the path cannot be reached) · **Area:** Central–Site Communication
> **Resolution (2026-07-23):** Decision = **delete** (option 1 below). Removed the
> `IntegrationCallRequest`/`IntegrationCallResponse` messages, `CommunicationService.RouteIntegrationCallAsync`,
> the `SiteCommunicationActor` receive block + `_integrationHandler` field + `LocalHandlerType.Integration`,
> and the four tests that covered them. `IntegrationTimeout` was **kept** — it is the live timeout for the
> Inbound API's `RouteTo*` verbs, which are the actual implementation of this "External → Central → Site →
> Central" pattern (design §4). The narrative comments that documented the exclusion (proto/mapper/dispatcher)
> were updated. Full solution build clean; Communication suite 634 green. This note is retained as the record.
## What
"Pattern 4: Integration Routing" — `CommunicationService.RouteIntegrationCallAsync` →
`SiteEnvelope(IntegrationCallRequest)` → `SiteCommunicationActor` → an integration handler — is
plumbed end to end but connected at neither end.
- **No producer.** `RouteIntegrationCallAsync` (`CommunicationService.cs`, "Pattern 4") has **zero
callers** in `src/` or `tests/`. It is the only one of `CommunicationService`'s command methods
with none.
- **No handler.** `SiteCommunicationActor` forwards to `_integrationHandler` when one is
registered, but `RegisterLocalHandler(LocalHandlerType.Integration, …)` appears **only** in
`SiteCommunicationActorTests.cs`. `AkkaHostedService` registers the other three handler types
(`Artifacts`, `EventLog`, `ParkedMessages`) and never this one.
So if anything ever did call it, the site would answer
`IntegrationCallResponse(Success: false, Error: "Integration handler not available")`
(`SiteCommunicationActor.cs`, Pattern 4) — and the two tests that exercise the path both register
the handler themselves first, which is why the suite has never noticed.
Do not confuse this with the **Inbound API**'s routed-site-script path, which is live, tested, and
uses different messages entirely. This is a separate, unused routing pattern that predates it.
## Why it is recorded rather than fixed
Found during the recon for the ClusterClient→gRPC transport migration
([`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`](../plans/2026-07-22-clusterclient-to-grpc-plan.md),
T0.2), which had to enumerate every command crossing the site↔central boundary. Of the **29**
command types, this is the one that is excluded: **28 migrate to the gRPC contract.**
Porting it would mean designing a proto contract, a `oneof` slot and round-trip mapper tests for a
verb no caller can invoke and no site can service — and enshrining it on a wire format whose
evolution rules are additive-only, so an unused RPC slot is permanent. Deleting it during a
transport migration would mix an unrelated behavioural change into a change whose whole value is
that behaviour is identical. Hence: excluded from the contract, behaviour untouched, decision
deferred to its own change.
## Decision needed
Either:
1. **Delete** — remove `RouteIntegrationCallAsync`, the `IntegrationCallRequest`/`Response`
messages, the `SiteCommunicationActor` receive block, `LocalHandlerType.Integration`, and the
three tests that cover them. This is the default if no consumer is planned.
2. **Wire** — register a real integration handler on site nodes and give the method a caller. This
only makes sense if there is a requirement it serves; none is recorded in
`docs/requirements/`.
Whichever is chosen, do it **before Phase 4** of the migration, since Phase 4 deletes the Akka
transport underneath this path. If it is still dead at that point, option 1 is forced.
## Filing
To be filed as a Gitea issue on `dohertj2/scadabridge` by the repo owner — this note is the
in-repo record of the finding and of the migration exclusion it justifies.
docs/operations/2026-07-16-secrets-clustered-master-key.md
+12
View File
@@ -1,5 +1,17 @@
# Secrets: Clustered Master-Key Posture (Central Pair)
> **Update 2026-08-07 (truth sweep):** two claims below are stale.
> (1) "there is no built-in cross-node replication today" — cross-node replication
> has since shipped: ScadaBridge adopted opt-in **SQL-Server hub replication** for
> the host secret store (commit `8e12f994`, "feat(secrets): opt-in SQL-Server hub
> replication for the host secret store"; `ZB.MOM.WW.Secrets` 0.2.x Replicator
> packages), covered by the shared library's clustered-secrets runbook
> (`scadaproj/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`).
> (2) the G-8 KEK-rotation runbook is no longer "not yet built" — it ships with the
> shared library at `scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`
> (lib 0.1.3, `Rewrap`/`rewrap-all`). The interim shared-volume posture below
> remains valid but is no longer the only option.
## Purpose
`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
docs/plans/2026-03-16-cluster-infrastructure-refinement-design.md
+1 -1
View File
@@ -2,7 +2,7 @@
**Date**: 2026-03-16
**Component**: Cluster Infrastructure (`docs/requirements/Component-ClusterInfrastructure.md`)
**Status**: Approved
**Status**: Approved — superseded in part: the keep-oldest SBR decision was replaced by the auto-down decision 2026-07-21 (`docs/plans/2026-07-21-auto-down-availability-decision.md`).
## Problem
docs/plans/2026-03-16-communication-layer-refinement-design.md
+1 -1
View File
@@ -2,7 +2,7 @@
**Date**: 2026-03-16
**Component**: Central–Site Communication (`docs/requirements/Component-Communication.md`)
**Status**: Approved
**Status**: Approved — transport decisions superseded by the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`); the "no buffering at central" decision still stands.
## Problem
docs/plans/2026-03-17-deploy-artifacts-remove-configdb.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{
"planPath": "docs/plans/2026-03-17-deploy-artifacts-remove-configdb.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [
{"id": 4, "subject": "Task 1: Create SiteExternalSystemRepository", "status": "pending"},
{"id": 5, "subject": "Task 2: Create SiteNotificationRepository", "status": "pending"},
{"id": 6, "subject": "Task 3: Add data connections to DeployArtifactsCommand", "status": "pending"},
{"id": 7, "subject": "Task 4: Wire site-local repositories into DI", "status": "pending", "blockedBy": [4, 5, 6]},
{"id": 8, "subject": "Task 5: Add Deploy Artifacts button to Sites admin page", "status": "pending", "blockedBy": [9]},
{"id": 9, "subject": "Task 6: Update ArtifactDeploymentService to include all artifact types", "status": "pending"},
{"id": 10, "subject": "Task 7: End-to-end test", "status": "pending", "blockedBy": [7, 8, 9]}
{"id": 4, "subject": "Task 1: Create SiteExternalSystemRepository", "status": "completed"},
{"id": 5, "subject": "Task 2: Create SiteNotificationRepository", "status": "completed", "note": "Shipped as planned, then deliberately removed 2026-07-10 (arch-review 08 §1.3 — excise vestigial site notification surface; notification delivery is central-only). The notification_lists / smtp_configurations tables survive on the site as deliberately-empty, unregistered (non-replicated) tables."},
{"id": 6, "subject": "Task 3: Add data connections to DeployArtifactsCommand", "status": "completed"},
{"id": 7, "subject": "Task 4: Wire site-local repositories into DI", "status": "completed", "blockedBy": [4, 5, 6]},
{"id": 8, "subject": "Task 5: Add Deploy Artifacts button to Sites admin page", "status": "completed", "blockedBy": [9]},
{"id": 9, "subject": "Task 6: Update ArtifactDeploymentService to include all artifact types", "status": "completed"},
{"id": 10, "subject": "Task 7: End-to-end test", "status": "completed", "blockedBy": [7, 8, 9]}
],
"lastUpdated": "2026-03-17T16:40:00Z"
"lastUpdated": "2026-08-01T00:00:00Z"
}
docs/plans/2026-03-17-management-service-cli.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{
"planPath": "docs/plans/2026-03-17-management-service-cli.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [
{"id": 11, "subject": "Task 1: Create ManagementService project and test project", "status": "pending"},
{"id": 12, "subject": "Task 2: Define management message contracts in Commons", "status": "pending"},
{"id": 13, "subject": "Task 3: Implement ManagementActor", "status": "pending", "blockedBy": [11, 12]},
{"id": 14, "subject": "Task 4: Register ManagementActor on Central with ClusterClientReceptionist", "status": "pending", "blockedBy": [13]},
{"id": 15, "subject": "Task 5: Create CLI project with ClusterClient scaffolding", "status": "pending", "blockedBy": [12]},
{"id": 16, "subject": "Task 6: Implement CLI command groups", "status": "pending", "blockedBy": [15]},
{"id": 17, "subject": "Task 7: Write ManagementActor unit tests", "status": "pending", "blockedBy": [13]},
{"id": 18, "subject": "Task 8: End-to-end integration test", "status": "pending", "blockedBy": [14, 16, 17]}
{"id": 11, "subject": "Task 1: Create ManagementService project and test project", "status": "completed"},
{"id": 12, "subject": "Task 2: Define management message contracts in Commons", "status": "completed"},
{"id": 13, "subject": "Task 3: Implement ManagementActor", "status": "completed", "blockedBy": [11, 12]},
{"id": 14, "subject": "Task 4: Register ManagementActor on Central with ClusterClientReceptionist", "status": "completed", "blockedBy": [13], "note": "OBSOLETE — never built; CLI shipped HTTP-only (ManagementHttpClient), receptionist deleted in the 2026-07-22 gRPC migration. Marked completed only to close the tracker."},
{"id": 15, "subject": "Task 5: Create CLI project with ClusterClient scaffolding", "status": "completed", "blockedBy": [12], "note": "OBSOLETE — never built; CLI shipped HTTP-only (ManagementHttpClient), receptionist deleted in the 2026-07-22 gRPC migration. The CLI project itself exists; only the ClusterClient scaffolding is obsolete."},
{"id": 16, "subject": "Task 6: Implement CLI command groups", "status": "completed", "blockedBy": [15]},
{"id": 17, "subject": "Task 7: Write ManagementActor unit tests", "status": "completed", "blockedBy": [13]},
{"id": 18, "subject": "Task 8: End-to-end integration test", "status": "completed", "blockedBy": [14, 16, 17]}
],
"lastUpdated": "2026-03-17T17:00:00Z"
"lastUpdated": "2026-08-01T00:00:00Z"
}
docs/plans/2026-03-21-grpc-streaming-channel.md.tasks.json
+14 -13
View File
@@ -1,18 +1,19 @@
{
"planPath": "docs/plans/2026-03-21-grpc-streaming-channel.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code — feature shipped and later extended by the full ClusterClient→gRPC migration (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md)",
"tasks": [
{"id": 0, "taskId": "1", "subject": "Task 0: Proto Definition & Stub Generation", "status": "pending"},
{"id": 1, "taskId": "2", "subject": "Task 1: Site Config — GrpcPort in NodeOptions", "status": "pending", "blockedBy": [0]},
{"id": 2, "taskId": "3", "subject": "Task 2: Site Entity — gRPC Address Fields", "status": "pending", "blockedBy": [0]},
{"id": 3, "taskId": "4", "subject": "Task 3: Site-Side gRPC Server — StreamRelayActor", "status": "pending", "blockedBy": [0]},
{"id": 4, "taskId": "5", "subject": "Task 4: Site-Side gRPC Server — SiteStreamGrpcServer", "status": "pending", "blockedBy": [3]},
{"id": 5, "taskId": "6", "subject": "Task 5: Switch Site Host to WebApplicationBuilder + gRPC", "status": "pending", "blockedBy": [4]},
{"id": 6, "taskId": "7", "subject": "Task 6: Central-Side gRPC Client", "status": "pending", "blockedBy": [0]},
{"id": 7, "taskId": "8", "subject": "Task 7: Update DebugStreamBridgeActor to Use gRPC", "status": "pending", "blockedBy": [6, 5]},
{"id": 8, "taskId": "9", "subject": "Task 8: Remove ClusterClient Streaming Path", "status": "pending", "blockedBy": [7]},
{"id": 9, "taskId": "10", "subject": "Task 9: Docker & End-to-End Integration Test", "status": "pending", "blockedBy": [5, 1, 2]},
{"id": 10, "taskId": "11", "subject": "Task 10: Documentation Updates", "status": "pending", "blockedBy": [9]},
{"id": 11, "taskId": "12", "subject": "Task 11: Final Guardrail Tests", "status": "pending", "blockedBy": [9]}
{"id": 0, "taskId": "1", "subject": "Task 0: Proto Definition & Stub Generation", "status": "completed"},
{"id": 1, "taskId": "2", "subject": "Task 1: Site Config — GrpcPort in NodeOptions", "status": "completed", "blockedBy": [0]},
{"id": 2, "taskId": "3", "subject": "Task 2: Site Entity — gRPC Address Fields", "status": "completed", "blockedBy": [0]},
{"id": 3, "taskId": "4", "subject": "Task 3: Site-Side gRPC Server — StreamRelayActor", "status": "completed", "blockedBy": [0]},
{"id": 4, "taskId": "5", "subject": "Task 4: Site-Side gRPC Server — SiteStreamGrpcServer", "status": "completed", "blockedBy": [3]},
{"id": 5, "taskId": "6", "subject": "Task 5: Switch Site Host to WebApplicationBuilder + gRPC", "status": "completed", "blockedBy": [4]},
{"id": 6, "taskId": "7", "subject": "Task 6: Central-Side gRPC Client", "status": "completed", "blockedBy": [0]},
{"id": 7, "taskId": "8", "subject": "Task 7: Update DebugStreamBridgeActor to Use gRPC", "status": "completed", "blockedBy": [6, 5]},
{"id": 8, "taskId": "9", "subject": "Task 8: Remove ClusterClient Streaming Path", "status": "completed", "blockedBy": [7]},
{"id": 9, "taskId": "10", "subject": "Task 9: Docker & End-to-End Integration Test", "status": "completed", "blockedBy": [5, 1, 2]},
{"id": 10, "taskId": "11", "subject": "Task 10: Documentation Updates", "status": "completed", "blockedBy": [9]},
{"id": 11, "taskId": "12", "subject": "Task 11: Final Guardrail Tests", "status": "completed", "blockedBy": [9]}
],
"lastUpdated": "2026-03-21T14:15:00Z"
"lastUpdated": "2026-08-01T00:00:00Z"
}
docs/plans/2026-03-22-primary-backup-data-connections.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{
"planPath": "docs/plans/2026-03-22-primary-backup-data-connections.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code — shipped (EF migration AddPrimaryBackupDataConnections; failover coverage in DataConnectionActorTests)",
"tasks": [
{"id": 1, "subject": "Task 1: Entity Model & Database Migration", "status": "pending"},
{"id": 2, "subject": "Task 2: Update CreateConnectionCommand & Manager Actor", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: DataConnectionActor Failover State Machine", "status": "pending", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: Failover Tests", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Health Reporting & Site Event Logging", "status": "pending", "blockedBy": [3]},
{"id": 6, "subject": "Task 6: Central UI Changes", "status": "pending", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: CLI, Management API, and Deployment", "status": "pending", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Documentation Updates", "status": "pending", "blockedBy": [3]}
{"id": 1, "subject": "Task 1: Entity Model & Database Migration", "status": "completed", "note": "EF migration AddPrimaryBackupDataConnections."},
{"id": 2, "subject": "Task 2: Update CreateConnectionCommand & Manager Actor", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: DataConnectionActor Failover State Machine", "status": "completed", "blockedBy": [1, 2]},
{"id": 4, "subject": "Task 4: Failover Tests", "status": "completed", "blockedBy": [3], "note": "Covered by DataConnectionActorTests."},
{"id": 5, "subject": "Task 5: Health Reporting & Site Event Logging", "status": "completed", "blockedBy": [3]},
{"id": 6, "subject": "Task 6: Central UI Changes", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: CLI, Management API, and Deployment", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Documentation Updates", "status": "completed", "blockedBy": [3]}
],
"lastUpdated": "2026-03-22T12:00:00Z"
"lastUpdated": "2026-08-01T00:00:00Z"
}
docs/plans/2026-03-23-treeview-component.md.tasks.json
+12 -11
View File
@@ -1,16 +1,17 @@
{
"planPath": "docs/plans/2026-03-23-treeview-component.md",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"tasks": [
{"id": 22, "subject": "Task 1: Create TreeView.razor — Core Rendering (R1-R4, R14)", "status": "pending"},
{"id": 23, "subject": "Task 2: Add Selection Support (R5)", "status": "pending", "blockedBy": [22]},
{"id": 24, "subject": "Task 3: Add Session Storage Persistence (R11)", "status": "pending", "blockedBy": [23]},
{"id": 25, "subject": "Task 4: Add ExpandAll, CollapseAll, RevealNode (R12, R13)", "status": "pending", "blockedBy": [24]},
{"id": 26, "subject": "Task 5: Add Context Menu (R15)", "status": "pending", "blockedBy": [25]},
{"id": 27, "subject": "Task 6: Add External Filtering Tests (R8)", "status": "pending", "blockedBy": [26]},
{"id": 28, "subject": "Task 7: Integrate TreeView into Data Connections Page", "status": "pending", "blockedBy": [27]},
{"id": 29, "subject": "Task 8: Integrate TreeView into Areas Page", "status": "pending", "blockedBy": [27]},
{"id": 30, "subject": "Task 9: Integrate TreeView into Instances Page", "status": "pending", "blockedBy": [27]},
{"id": 31, "subject": "Task 10: Full Build Verification", "status": "pending", "blockedBy": [28, 29, 30]}
{"id": 22, "subject": "Task 1: Create TreeView.razor — Core Rendering (R1-R4, R14)", "status": "completed"},
{"id": 23, "subject": "Task 2: Add Selection Support (R5)", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 3: Add Session Storage Persistence (R11)", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 4: Add ExpandAll, CollapseAll, RevealNode (R12, R13)", "status": "completed", "blockedBy": [24]},
{"id": 26, "subject": "Task 5: Add Context Menu (R15)", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 6: Add External Filtering Tests (R8)", "status": "completed", "blockedBy": [26]},
{"id": 28, "subject": "Task 7: Integrate TreeView into Data Connections Page", "status": "completed", "blockedBy": [27]},
{"id": 29, "subject": "Task 8: Integrate TreeView into Areas Page", "status": "completed", "blockedBy": [27], "note": "OBSOLETE — those pages never existed; Site/Area/Instance render in one tree on Pages/Deployment/Topology.razor (2026-05-11 topology design). Marked completed only to close the tracker."},
{"id": 30, "subject": "Task 9: Integrate TreeView into Instances Page", "status": "completed", "blockedBy": [27], "note": "OBSOLETE — those pages never existed; Site/Area/Instance render in one tree on Pages/Deployment/Topology.razor (2026-05-11 topology design). Marked completed only to close the tracker."},
{"id": 31, "subject": "Task 10: Full Build Verification", "status": "completed", "blockedBy": [28, 29, 30]}
],
"lastUpdated": "2026-03-23T00:00:00Z"
"lastUpdated": "2026-08-01T00:00:00Z"
}
docs/plans/2026-05-11-deployment-topology-page-design.md
+3 -3
View File
@@ -138,9 +138,9 @@ Areas can be moved freely (subject to validation). Templates are different becau
- `AreaService.UpdateAreaAsync` (stays name-only)
- `InstanceService` lifecycle methods (already used by current Instances page)
### CLI / ManagementService parity (optional follow-up)
- Add `MoveAreaCommand` message + `ManagementService` handler that wraps `MoveAreaAsync`.
- Add CLI: `cli area move --id X --parent-id Y --username … --password …` (omit `--parent-id` to move to site root).
### CLI / ManagementService parity (optional follow-up) — **DONE 2026-08-01**
- ~~Add `MoveAreaCommand` message + `ManagementService` handler that wraps `MoveAreaAsync`.~~ Shipped: `MoveAreaCommand(int AreaId, int? NewParentAreaId)` in `Commons/Messages/Management/SiteCommands.cs`, dispatched by `ManagementActor.HandleMoveArea` (delegates to `AreaService.MoveAreaAsync`; failures surface as the standard curated failure response), gated any-of `[Designer, Deployer]` like the other area mutations.
- ~~Add CLI: `cli area move --id X --parent-id Y …` (omit `--parent-id` to move to site root).~~ Shipped as **`scadabridge site area move --id X [--parent-id Y]`** — the area verbs live under the existing `site area` group, not at the CLI root, so the verb was placed alongside `site area create|update|delete` rather than introducing a second top-level spelling.
Not strictly required to ship the UI page, but worth doing for parity with how the rest of the app exposes admin ops.
docs/plans/2026-05-11-templates-folder-hierarchy.md
+10 -1
View File
@@ -2180,6 +2180,15 @@ Expected: image rebuilds, 5-container cluster starts.
**Step 3: Manual smoke checklist**
> **RETIRED 2026-08-01 — never run; unrunnable as written and superseded by delivered coverage.**
> Two steps (drag a template onto "Dev"; drag "Dev" onto "Sub") reference folder drag-drop, which was
> **permanently deferred** by the M9 decision (menu-based reorder shipped instead — see the deferred-work
> register's Resolved row 18), so the checklist cannot pass as written. The remaining behaviors are
> covered by the CentralUI bUnit suites (folder CRUD/context menus/cycle guard/TreeView reveal +
> sessionStorage expansion) delivered through M9/M10, and the management folder commands gained full CLI
> parity on 2026-08-01 (`template folder list|create|rename|move|reorder|delete`, commit `88638d77`).
> Do not run or refresh this checklist.
Open http://localhost:9000/design/templates (login `multi-role` / `password`). Verify:
- [ ] Existing templates appear at root (no folder).
@@ -2202,7 +2211,7 @@ Open http://localhost:9000/design/templates (login `multi-role` / `password`). V
## Out of scope (per design)
- CLI commands for folder operations (Management Service contracts now exist; CLI follows in a future plan).
- ~~CLI commands for folder operations (Management Service contracts now exist; CLI follows in a future plan).~~ **DONE 2026-08-01** — shipped as `scadabridge template folder list|create|rename|move|reorder|delete`, mapping 1:1 onto `ListTemplateFolders` / `CreateTemplateFolder` / `RenameTemplateFolder` / `MoveTemplateFolder` / `ReorderTemplateFolder` / `DeleteTemplateFolder`. Omitting `--parent-id` on create/move targets the tree root; `reorder --direction` takes the lowercase literals `up` / `down`.
- Tree search / filter input.
- Sibling reordering via drag-drop (alphabetical sort is fixed).
- Root-area context menu (right-click in empty tree space).
docs/plans/2026-05-11-templates-folder-hierarchy.md.tasks.json
+25 -24
View File
@@ -1,29 +1,30 @@
{
"planPath": "docs/plans/2026-05-11-templates-folder-hierarchy.md",
"tasks": [
{"id": 7, "subject": "Task 0: Confirm baseline + create work branch", "status": "pending"},
{"id": 8, "subject": "Task 1: Add TemplateFolder entity + Template.FolderId", "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 2: EF configuration for TemplateFolder + Template.FolderId", "status": "pending", "blockedBy": [8]},
{"id": 10, "subject": "Task 3: Generate EF migration AddTemplateFolders", "status": "pending", "blockedBy": [9]},
{"id": 11, "subject": "Task 4: Repository methods for TemplateFolder", "status": "pending", "blockedBy": [10]},
{"id": 12, "subject": "Task 5: TemplateFolderService.CreateFolderAsync (TDD)", "status": "pending", "blockedBy": [11]},
{"id": 13, "subject": "Task 6: TemplateFolderService.RenameFolderAsync", "status": "pending", "blockedBy": [12]},
{"id": 14, "subject": "Task 7: TemplateFolderService.MoveFolderAsync with cycle detection", "status": "pending", "blockedBy": [13]},
{"id": 15, "subject": "Task 8: TemplateFolderService.DeleteFolderAsync (non-empty check)", "status": "pending", "blockedBy": [14]},
{"id": 16, "subject": "Task 9: TemplateService.MoveTemplateAsync", "status": "pending", "blockedBy": [11]},
{"id": 17, "subject": "Task 10: DI registration for TemplateFolderService", "status": "pending", "blockedBy": [15, 16]},
{"id": 18, "subject": "Task 11: Management command records for TemplateFolder", "status": "pending", "blockedBy": [17]},
{"id": 19, "subject": "Task 12: ManagementActor authorization + handlers", "status": "pending", "blockedBy": [18]},
{"id": 20, "subject": "Task 13: Templates.razor — load folders alongside templates", "status": "pending", "blockedBy": [17]},
{"id": 21, "subject": "Task 14: Build new TmplNode tree model", "status": "pending", "blockedBy": [20]},
{"id": 22, "subject": "Task 15: Split-pane layout + new TreeView wiring", "status": "pending", "blockedBy": [21]},
{"id": 23, "subject": "Task 16: Per-kind context menus", "status": "pending", "blockedBy": [22]},
{"id": 24, "subject": "Task 17: New-folder, new-template, move-template dialogs", "status": "pending", "blockedBy": [23]},
{"id": 25, "subject": "Task 18: Drag-drop reorganization", "status": "pending", "blockedBy": [24]},
{"id": 26, "subject": "Task 19: Deep-link reveal on load", "status": "pending", "blockedBy": [22]},
{"id": 27, "subject": "Task 20: bUnit tests for the new page", "status": "pending", "blockedBy": [22]},
{"id": 28, "subject": "Task 21: Documentation updates", "status": "pending", "blockedBy": [25, 26, 27]},
{"id": 29, "subject": "Task 22: Final smoke + green-suite check", "status": "pending", "blockedBy": [25, 26, 27, 28]}
{"id": 7, "subject": "Task 0: Confirm baseline + create work branch", "status": "completed"},
{"id": 8, "subject": "Task 1: Add TemplateFolder entity + Template.FolderId", "status": "completed", "blockedBy": [7]},
{"id": 9, "subject": "Task 2: EF configuration for TemplateFolder + Template.FolderId", "status": "completed", "blockedBy": [8]},
{"id": 10, "subject": "Task 3: Generate EF migration AddTemplateFolders", "status": "completed", "blockedBy": [9]},
{"id": 11, "subject": "Task 4: Repository methods for TemplateFolder", "status": "completed", "blockedBy": [10]},
{"id": 12, "subject": "Task 5: TemplateFolderService.CreateFolderAsync (TDD)", "status": "completed", "blockedBy": [11]},
{"id": 13, "subject": "Task 6: TemplateFolderService.RenameFolderAsync", "status": "completed", "blockedBy": [12]},
{"id": 14, "subject": "Task 7: TemplateFolderService.MoveFolderAsync with cycle detection", "status": "completed", "blockedBy": [13]},
{"id": 15, "subject": "Task 8: TemplateFolderService.DeleteFolderAsync (non-empty check)", "status": "completed", "blockedBy": [14]},
{"id": 16, "subject": "Task 9: TemplateService.MoveTemplateAsync", "status": "completed", "blockedBy": [11]},
{"id": 17, "subject": "Task 10: DI registration for TemplateFolderService", "status": "completed", "blockedBy": [15, 16]},
{"id": 18, "subject": "Task 11: Management command records for TemplateFolder", "status": "completed", "blockedBy": [17]},
{"id": 19, "subject": "Task 12: ManagementActor authorization + handlers", "status": "completed", "blockedBy": [18]},
{"id": 20, "subject": "Task 13: Templates.razor — load folders alongside templates", "status": "completed", "blockedBy": [17]},
{"id": 21, "subject": "Task 14: Build new TmplNode tree model", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 15: Split-pane layout + new TreeView wiring", "status": "completed", "blockedBy": [21]},
{"id": 23, "subject": "Task 16: Per-kind context menus", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 17: New-folder, new-template, move-template dialogs", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 18: Drag-drop reorganization", "status": "completed", "blockedBy": [24], "notes": "DROPPED — superseded by M9 menu-based reorder (commits e3bc19c6/314c7dea) + MoveFolderDialog; no DnD anywhere in CentralUI"},
{"id": 26, "subject": "Task 19: Deep-link reveal on load", "status": "completed", "blockedBy": [22], "notes": "OBSOLETE — /design/templates/{id} is TemplateEdit's own route; two-page split makes in-tree reveal moot"},
{"id": 27, "subject": "Task 20: bUnit tests for the new page", "status": "completed", "blockedBy": [22]},
{"id": 28, "subject": "Task 21: Documentation updates", "status": "completed", "blockedBy": [25, 26, 27]},
{"id": 29, "subject": "Task 22: Final smoke + green-suite check", "status": "completed", "blockedBy": [25, 26, 27, 28]}
],
"lastUpdated": "2026-05-11"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-12-derive-on-compose-status.md
+6
View File
@@ -1,5 +1,11 @@
# Derive-on-compose: implementation status
> **2026-08-01:** The "Still to verify" items below are satisfied — the
> Phase-3 migration ships and auto-applies, and
> `2026-05-18-contained-template-names-design.md` records it applied and
> browser-verified on the dev cluster. The "How to resume" section is
> historical.
> **For Claude resuming later:** All nine phases are implemented. This
> file is the change-record for the work, not a plan. See the companion
> design doc `2026-05-12-derive-on-compose-design.md` for rationale.
docs/plans/2026-05-12-opcua-config-model.md.tasks.json
+16 -15
View File
@@ -1,20 +1,21 @@
{
"planPath": "docs/plans/2026-05-12-opcua-config-model.md",
"tasks": [
{"id": 45, "subject": "Task 1: Create OPC UA config POCOs + ValidationCategory.ConnectionConfig", "status": "pending"},
{"id": 46, "subject": "Task 2: TDD failing tests for OpcUaEndpointConfigSerializer", "status": "pending", "blockedBy": [45]},
{"id": 47, "subject": "Task 3: Implement OpcUaEndpointConfigSerializer", "status": "pending", "blockedBy": [46]},
{"id": 48, "subject": "Task 4: TDD failing tests for OpcUaEndpointConfigValidator", "status": "pending", "blockedBy": [45]},
{"id": 49, "subject": "Task 5: Implement OpcUaEndpointConfigValidator", "status": "pending", "blockedBy": [48]},
{"id": 50, "subject": "Task 6: Refactor OpcUaDataConnection.ConnectAsync to use FromFlatDict", "status": "pending", "blockedBy": [47]},
{"id": 51, "subject": "Task 7: Refactor DeploymentManagerActor.EnsureDclConnections", "status": "pending", "blockedBy": [47]},
{"id": 52, "subject": "Task 8: TDD failing bUnit tests for OpcUaEndpointEditor", "status": "pending", "blockedBy": [45, 49]},
{"id": 53, "subject": "Task 9: Implement OpcUaEndpointEditor.razor", "status": "pending", "blockedBy": [52]},
{"id": 54, "subject": "Task 10: TDD failing bUnit tests for DataConnectionForm refactor", "status": "pending", "blockedBy": [47, 49]},
{"id": 55, "subject": "Task 11: Refactor DataConnectionForm.razor", "status": "pending", "blockedBy": [53, 54]},
{"id": 56, "subject": "Task 12: Solution build + all test suites green", "status": "pending", "blockedBy": [50, 51, 55]},
{"id": 57, "subject": "Task 13: Docker deploy + browser smoke", "status": "pending", "blockedBy": [56]},
{"id": 58, "subject": "Task 14: Push to origin", "status": "pending", "blockedBy": [57]}
{"id": 45, "subject": "Task 1: Create OPC UA config POCOs + ValidationCategory.ConnectionConfig", "status": "completed"},
{"id": 46, "subject": "Task 2: TDD failing tests for OpcUaEndpointConfigSerializer", "status": "completed", "blockedBy": [45]},
{"id": 47, "subject": "Task 3: Implement OpcUaEndpointConfigSerializer", "status": "completed", "blockedBy": [46]},
{"id": 48, "subject": "Task 4: TDD failing tests for OpcUaEndpointConfigValidator", "status": "completed", "blockedBy": [45]},
{"id": 49, "subject": "Task 5: Implement OpcUaEndpointConfigValidator", "status": "completed", "blockedBy": [48]},
{"id": 50, "subject": "Task 6: Refactor OpcUaDataConnection.ConnectAsync to use FromFlatDict", "status": "completed", "blockedBy": [47]},
{"id": 51, "subject": "Task 7: Refactor DeploymentManagerActor.EnsureDclConnections", "status": "completed", "blockedBy": [47]},
{"id": 52, "subject": "Task 8: TDD failing bUnit tests for OpcUaEndpointEditor", "status": "completed", "blockedBy": [45, 49]},
{"id": 53, "subject": "Task 9: Implement OpcUaEndpointEditor.razor", "status": "completed", "blockedBy": [52]},
{"id": 54, "subject": "Task 10: TDD failing bUnit tests for DataConnectionForm refactor", "status": "completed", "blockedBy": [47, 49]},
{"id": 55, "subject": "Task 11: Refactor DataConnectionForm.razor", "status": "completed", "blockedBy": [53, 54]},
{"id": 56, "subject": "Task 12: Solution build + all test suites green", "status": "completed", "blockedBy": [50, 51, 55]},
{"id": 57, "subject": "Task 13: Docker deploy + browser smoke", "status": "completed", "blockedBy": [56]},
{"id": 58, "subject": "Task 14: Push to origin", "status": "completed", "blockedBy": [57]}
],
"lastUpdated": "2026-05-12T04:33:33Z"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-12-ui-audit.md
+2
View File
@@ -5,6 +5,8 @@
**Scope:** All Razor pages, layout, and shared components in `src/ZB.MOM.WW.ScadaBridge.CentralUI`.
**Reference pattern:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Admin/Sites.razor` — 2-column responsive card grid, header flex row, kebab menus, search filter, Bootstrap collapse for noisy details, `@key=` on iterated cards, "No X match the filter." and empty-state CTAs.
> **2026-08-01: fully superseded** — every cross-cutting recommendation was either shipped or explicitly scoped into the M10 UI/UX platform plan (`2026-06-18`) and KPI History (#26); retained for reference only.
## Constraints (recap)
- Blazor Server + Bootstrap 5 only. **No third-party component frameworks** (no MudBlazor / Radzen / Blazorise / Syncfusion).
docs/plans/2026-05-16-expression-trigger.md.tasks.json
+7 -6
View File
@@ -1,11 +1,12 @@
{
"planPath": "docs/plans/2026-05-16-expression-trigger.md",
"tasks": [
{"id": 25, "subject": "Task 1: Trigger model + codecs", "status": "pending"},
{"id": 26, "subject": "Task 2: Runtime expression evaluation", "status": "pending", "blockedBy": [25]},
{"id": 27, "subject": "Task 3: Trigger editor panels", "status": "pending", "blockedBy": [25]},
{"id": 28, "subject": "Task 4: Pre-deployment validation", "status": "pending", "blockedBy": [25, 26]},
{"id": 29, "subject": "Task 5: Build, deploy, verify", "status": "pending", "blockedBy": [25, 26, 27, 28]}
{"id": 25, "subject": "Task 1: Trigger model + codecs", "status": "completed"},
{"id": 26, "subject": "Task 2: Runtime expression evaluation", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 3: Trigger editor panels", "status": "completed", "blockedBy": [25]},
{"id": 28, "subject": "Task 4: Pre-deployment validation", "status": "completed", "blockedBy": [25, 26]},
{"id": 29, "subject": "Task 5: Build, deploy, verify", "status": "completed", "blockedBy": [25, 26, 27, 28]}
],
"lastUpdated": "2026-05-16"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-18-notification-outbox.md.tasks.json
+13 -12
View File
@@ -1,17 +1,18 @@
{
"planPath": "docs/plans/2026-05-18-notification-outbox.md",
"tasks": [
{"id": 7, "subject": "Task 1: Create Component-NotificationOutbox.md", "status": "pending"},
{"id": 8, "subject": "Task 2: Revise Component-NotificationService.md", "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 3: Revise Component-StoreAndForward.md", "status": "pending", "blockedBy": [7]},
{"id": 10, "subject": "Task 4: Revise Component-HealthMonitoring.md", "status": "pending", "blockedBy": [7]},
{"id": 11, "subject": "Task 5: Revise Component-SiteEventLogging.md", "status": "pending", "blockedBy": [7]},
{"id": 12, "subject": "Task 6: Revise Component-Communication.md", "status": "pending", "blockedBy": [7]},
{"id": 13, "subject": "Task 7: Revise Component-CentralUI.md", "status": "pending", "blockedBy": [7]},
{"id": 14, "subject": "Task 8: Revise Component-ConfigurationDatabase.md and Component-Commons.md", "status": "pending", "blockedBy": [7]},
{"id": 15, "subject": "Task 9: Update README.md", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 10: Update CLAUDE.md", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 17, "subject": "Task 11: Cross-reference consistency sweep", "status": "pending", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}
{"id": 7, "subject": "Task 1: Create Component-NotificationOutbox.md", "status": "completed"},
{"id": 8, "subject": "Task 2: Revise Component-NotificationService.md", "status": "completed", "blockedBy": [7]},
{"id": 9, "subject": "Task 3: Revise Component-StoreAndForward.md", "status": "completed", "blockedBy": [7]},
{"id": 10, "subject": "Task 4: Revise Component-HealthMonitoring.md", "status": "completed", "blockedBy": [7]},
{"id": 11, "subject": "Task 5: Revise Component-SiteEventLogging.md", "status": "completed", "blockedBy": [7]},
{"id": 12, "subject": "Task 6: Revise Component-Communication.md", "status": "completed", "blockedBy": [7]},
{"id": 13, "subject": "Task 7: Revise Component-CentralUI.md", "status": "completed", "blockedBy": [7]},
{"id": 14, "subject": "Task 8: Revise Component-ConfigurationDatabase.md and Component-Commons.md", "status": "completed", "blockedBy": [7]},
{"id": 15, "subject": "Task 9: Update README.md", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 10: Update CLAUDE.md", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 17, "subject": "Task 11: Cross-reference consistency sweep", "status": "completed", "blockedBy": [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}
],
"lastUpdated": "2026-05-18"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-19-cached-call-tracking.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{
"planPath": "docs/plans/2026-05-19-cached-call-tracking.md",
"tasks": [
{"id": 6, "subject": "Task 1: Create Site Call Audit component doc", "status": "pending"},
{"id": 7, "subject": "Task 2: Add tracking contracts to Commons", "status": "pending", "blockedBy": [6]},
{"id": 8, "subject": "Task 3: Update Store-and-Forward doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 9, "subject": "Task 4: Update External System Gateway doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 10, "subject": "Task 5: Update Site Runtime Script Runtime API", "status": "pending", "blockedBy": [6, 7]},
{"id": 11, "subject": "Task 6: Update Communication doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 12, "subject": "Task 7: Update Configuration Database doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 13, "subject": "Task 8: Update Central UI doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 14, "subject": "Task 9: Update Health Monitoring doc", "status": "pending", "blockedBy": [6, 7]},
{"id": 15, "subject": "Task 10: Note shared model in notification docs", "status": "pending", "blockedBy": [6, 7]},
{"id": 16, "subject": "Task 11: Update README component table", "status": "pending", "blockedBy": [6]},
{"id": 17, "subject": "Task 12: Update CLAUDE.md", "status": "pending", "blockedBy": [6]},
{"id": 18, "subject": "Task 13: Final cross-reference consistency pass", "status": "pending", "blockedBy": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]}
{"id": 6, "subject": "Task 1: Create Site Call Audit component doc", "status": "completed"},
{"id": 7, "subject": "Task 2: Add tracking contracts to Commons", "status": "completed", "blockedBy": [6]},
{"id": 8, "subject": "Task 3: Update Store-and-Forward doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 9, "subject": "Task 4: Update External System Gateway doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 10, "subject": "Task 5: Update Site Runtime Script Runtime API", "status": "completed", "blockedBy": [6, 7]},
{"id": 11, "subject": "Task 6: Update Communication doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 12, "subject": "Task 7: Update Configuration Database doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 13, "subject": "Task 8: Update Central UI doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 14, "subject": "Task 9: Update Health Monitoring doc", "status": "completed", "blockedBy": [6, 7]},
{"id": 15, "subject": "Task 10: Note shared model in notification docs", "status": "completed", "blockedBy": [6, 7]},
{"id": 16, "subject": "Task 11: Update README component table", "status": "completed", "blockedBy": [6]},
{"id": 17, "subject": "Task 12: Update CLAUDE.md", "status": "completed", "blockedBy": [6]},
{"id": 18, "subject": "Task 13: Final cross-reference consistency pass", "status": "completed", "blockedBy": [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]}
],
"lastUpdated": "2026-05-19"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-19-notification-outbox-implementation.md.tasks.json
+28 -27
View File
@@ -1,32 +1,33 @@
{
"planPath": "docs/plans/2026-05-19-notification-outbox-implementation.md",
"tasks": [
{"id": 18, "subject": "Task 1: Notification enums", "status": "pending"},
{"id": 19, "subject": "Task 2: Notification entity POCO", "status": "pending", "blockedBy": [18]},
{"id": 20, "subject": "Task 3: Type field on NotificationList", "status": "pending", "blockedBy": [19]},
{"id": 21, "subject": "Task 4: Notification EF configuration + DbSet", "status": "pending", "blockedBy": [20]},
{"id": 22, "subject": "Task 5: NotificationOutbox repository", "status": "pending", "blockedBy": [21]},
{"id": 23, "subject": "Task 6: EF migration AddNotificationsTable", "status": "pending", "blockedBy": [22]},
{"id": 24, "subject": "Task 7: Site/central notification message contracts", "status": "pending", "blockedBy": [23]},
{"id": 25, "subject": "Task 8: Outbox query/action contracts", "status": "pending", "blockedBy": [24]},
{"id": 26, "subject": "Task 9: Scaffold ZB.MOM.WW.ScadaBridge.NotificationOutbox project", "status": "pending", "blockedBy": [25]},
{"id": 27, "subject": "Task 10: NotificationOutboxOptions", "status": "pending", "blockedBy": [26]},
{"id": 28, "subject": "Task 11: Delivery adapter abstraction", "status": "pending", "blockedBy": [27]},
{"id": 29, "subject": "Task 12: Email delivery adapter", "status": "pending", "blockedBy": [28]},
{"id": 30, "subject": "Task 13: NotificationOutboxActor ingest", "status": "pending", "blockedBy": [29]},
{"id": 31, "subject": "Task 14: Dispatcher loop", "status": "pending", "blockedBy": [30]},
{"id": 32, "subject": "Task 15: Query, retry, discard, KPI handlers", "status": "pending", "blockedBy": [31]},
{"id": 33, "subject": "Task 16: Daily purge job", "status": "pending", "blockedBy": [32]},
{"id": 34, "subject": "Task 17: AddNotificationOutbox DI extension", "status": "pending", "blockedBy": [33]},
{"id": 35, "subject": "Task 18: Retarget site S&F notification handler to central", "status": "pending", "blockedBy": [34]},
{"id": 36, "subject": "Task 19: Async Notify.Send + Notify.Status", "status": "pending", "blockedBy": [35]},
{"id": 37, "subject": "Task 20: Central ingest routing", "status": "pending", "blockedBy": [36]},
{"id": 38, "subject": "Task 21: Host registration + appsettings", "status": "pending", "blockedBy": [37]},
{"id": 39, "subject": "Task 22: CommunicationService outbox methods", "status": "pending", "blockedBy": [38]},
{"id": 40, "subject": "Task 23: Notification Outbox Blazor page", "status": "pending", "blockedBy": [39]},
{"id": 41, "subject": "Task 24: Health dashboard outbox KPI tiles", "status": "pending", "blockedBy": [40]},
{"id": 42, "subject": "Task 25: End-to-end integration test", "status": "pending", "blockedBy": [41]},
{"id": 43, "subject": "Task 26: Full build + suite verification", "status": "pending", "blockedBy": [42]}
{"id": 18, "subject": "Task 1: Notification enums", "status": "completed"},
{"id": 19, "subject": "Task 2: Notification entity POCO", "status": "completed", "blockedBy": [18]},
{"id": 20, "subject": "Task 3: Type field on NotificationList", "status": "completed", "blockedBy": [19]},
{"id": 21, "subject": "Task 4: Notification EF configuration + DbSet", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 5: NotificationOutbox repository", "status": "completed", "blockedBy": [21]},
{"id": 23, "subject": "Task 6: EF migration AddNotificationsTable", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "Task 7: Site/central notification message contracts", "status": "completed", "blockedBy": [23]},
{"id": 25, "subject": "Task 8: Outbox query/action contracts", "status": "completed", "blockedBy": [24]},
{"id": 26, "subject": "Task 9: Scaffold ZB.MOM.WW.ScadaBridge.NotificationOutbox project", "status": "completed", "blockedBy": [25]},
{"id": 27, "subject": "Task 10: NotificationOutboxOptions", "status": "completed", "blockedBy": [26]},
{"id": 28, "subject": "Task 11: Delivery adapter abstraction", "status": "completed", "blockedBy": [27]},
{"id": 29, "subject": "Task 12: Email delivery adapter", "status": "completed", "blockedBy": [28]},
{"id": 30, "subject": "Task 13: NotificationOutboxActor ingest", "status": "completed", "blockedBy": [29]},
{"id": 31, "subject": "Task 14: Dispatcher loop", "status": "completed", "blockedBy": [30]},
{"id": 32, "subject": "Task 15: Query, retry, discard, KPI handlers", "status": "completed", "blockedBy": [31]},
{"id": 33, "subject": "Task 16: Daily purge job", "status": "completed", "blockedBy": [32]},
{"id": 34, "subject": "Task 17: AddNotificationOutbox DI extension", "status": "completed", "blockedBy": [33]},
{"id": 35, "subject": "Task 18: Retarget site S&F notification handler to central", "status": "completed", "blockedBy": [34]},
{"id": 36, "subject": "Task 19: Async Notify.Send + Notify.Status", "status": "completed", "blockedBy": [35]},
{"id": 37, "subject": "Task 20: Central ingest routing", "status": "completed", "blockedBy": [36]},
{"id": 38, "subject": "Task 21: Host registration + appsettings", "status": "completed", "blockedBy": [37]},
{"id": 39, "subject": "Task 22: CommunicationService outbox methods", "status": "completed", "blockedBy": [38]},
{"id": 40, "subject": "Task 23: Notification Outbox Blazor page", "status": "completed", "blockedBy": [39]},
{"id": 41, "subject": "Task 24: Health dashboard outbox KPI tiles", "status": "completed", "blockedBy": [40]},
{"id": 42, "subject": "Task 25: End-to-end integration test", "status": "completed", "blockedBy": [41]},
{"id": 43, "subject": "Task 26: Full build + suite verification", "status": "completed", "blockedBy": [42]}
],
"lastUpdated": "2026-05-19"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-20-audit-log-code-roadmap.md
+2
View File
@@ -23,6 +23,8 @@
> for site roles, with `NoOpSiteStreamAuditClient` retained only for central/test
> composition roots; and `AuditLogQueryFilter` is now multi-value per dimension.
>
> **Transport wording below is stale:** the ClusterClient-based push described throughout this roadmap was replaced by gRPC in the 2026-07-22 ClusterClient→gRPC migration, and `ClusterClientSiteAuditClient` is now `SiteCommunicationAuditClient` (commit `63c16d69`).
>
> **For Claude:** REQUIRED SUB-SKILL FLOW per milestone: `brainstorming` → `writing-plans` → `subagent-driven-development`. Use `docs/requirements/Component-AuditLog.md` + `alog.md` as the spec; this document is the roadmap that sequences milestones and locks acceptance criteria for each. **M1 carries full TDD-level task detail; M2–M8 are milestone-shape detail and will be expanded into bite-sized plans by their own writing-plans pass when their turn comes.**
**Goal:** Implement central component #23 Audit Log — append-only forensic + operational record across every script-trust-boundary action — into the existing ScadaBridge codebase.
docs/plans/2026-05-20-auditlog-m1-foundation.md.tasks.json
+20 -19
View File
@@ -1,24 +1,25 @@
{
"planPath": "docs/plans/2026-05-20-auditlog-m1-foundation.md",
"tasks": [
{"id": "A1", "subject": "Bundle A T1: Add audit enums (Channel, Kind, Status, ForwardState)", "status": "pending"},
{"id": "A2", "subject": "Bundle A T2: Add AuditEvent record", "status": "pending", "blockedBy": ["A1"]},
{"id": "A3", "subject": "Bundle A T3: Add IAuditWriter + ICentralAuditWriter", "status": "pending", "blockedBy": ["A2"]},
{"id": "A4", "subject": "Bundle A T4: Add audit telemetry + pull message DTOs", "status": "pending", "blockedBy": ["A2"]},
{"id": "A-rev", "subject": "Bundle A combined spec+quality review", "status": "pending", "blockedBy": ["A1", "A2", "A3", "A4"]},
{"id": "B5", "subject": "Bundle B T5: ScadaBridgeDbContext.AuditLogs + IEntityTypeConfiguration<AuditEvent> with five named indexes", "status": "pending", "blockedBy": ["A-rev"]},
{"id": "B-rev", "subject": "Bundle B review", "status": "pending", "blockedBy": ["B5"]},
{"id": "C67", "subject": "Bundle C T6+T7: AddAuditLogTable migration (partition fn/scheme/table/indexes) + DB roles, with infra/mssql integration tests", "status": "pending", "blockedBy": ["B-rev"]},
{"id": "C-rev", "subject": "Bundle C review", "status": "pending", "blockedBy": ["C67"]},
{"id": "D8", "subject": "Bundle D T8: IAuditLogRepository + EF implementation + DI registration", "status": "pending", "blockedBy": ["C-rev"]},
{"id": "D-rev", "subject": "Bundle D review", "status": "pending", "blockedBy": ["D8"]},
{"id": "E10", "subject": "Bundle E T10: Scaffold src/ZB.MOM.WW.ScadaBridge.AuditLog/ project + slnx entries", "status": "pending", "blockedBy": ["D-rev"]},
{"id": "E9", "subject": "Bundle E T9: AuditLogOptions + validator", "status": "pending", "blockedBy": ["E10"]},
{"id": "E-rev", "subject": "Bundle E review", "status": "pending", "blockedBy": ["E10", "E9"]},
{"id": "F11", "subject": "Bundle F T11 (controller-direct): Register ZB.MOM.WW.ScadaBridge.AuditLog in Component-Host.md + README confirm", "status": "pending", "blockedBy": ["E-rev"]},
{"id": "FINAL-rev", "subject": "Final cross-bundle review over the whole M1 branch", "status": "pending", "blockedBy": ["F11"]},
{"id": "MERGE", "subject": "Verify gate: full solution dotnet test green, then merge --no-ff to main", "status": "pending", "blockedBy": ["FINAL-rev"]},
{"id": "ROADMAP", "subject": "Update downstream M2-M8 sections of roadmap with realities learned in M1", "status": "pending", "blockedBy": ["MERGE"]}
{"id": "A1", "subject": "Bundle A T1: Add audit enums (Channel, Kind, Status, ForwardState)", "status": "completed"},
{"id": "A2", "subject": "Bundle A T2: Add AuditEvent record", "status": "completed", "blockedBy": ["A1"]},
{"id": "A3", "subject": "Bundle A T3: Add IAuditWriter + ICentralAuditWriter", "status": "completed", "blockedBy": ["A2"]},
{"id": "A4", "subject": "Bundle A T4: Add audit telemetry + pull message DTOs", "status": "completed", "blockedBy": ["A2"]},
{"id": "A-rev", "subject": "Bundle A combined spec+quality review", "status": "completed", "blockedBy": ["A1", "A2", "A3", "A4"]},
{"id": "B5", "subject": "Bundle B T5: ScadaBridgeDbContext.AuditLogs + IEntityTypeConfiguration<AuditEvent> with five named indexes", "status": "completed", "blockedBy": ["A-rev"]},
{"id": "B-rev", "subject": "Bundle B review", "status": "completed", "blockedBy": ["B5"]},
{"id": "C67", "subject": "Bundle C T6+T7: AddAuditLogTable migration (partition fn/scheme/table/indexes) + DB roles, with infra/mssql integration tests", "status": "completed", "blockedBy": ["B-rev"]},
{"id": "C-rev", "subject": "Bundle C review", "status": "completed", "blockedBy": ["C67"]},
{"id": "D8", "subject": "Bundle D T8: IAuditLogRepository + EF implementation + DI registration", "status": "completed", "blockedBy": ["C-rev"]},
{"id": "D-rev", "subject": "Bundle D review", "status": "completed", "blockedBy": ["D8"]},
{"id": "E10", "subject": "Bundle E T10: Scaffold src/ZB.MOM.WW.ScadaBridge.AuditLog/ project + slnx entries", "status": "completed", "blockedBy": ["D-rev"]},
{"id": "E9", "subject": "Bundle E T9: AuditLogOptions + validator", "status": "completed", "blockedBy": ["E10"]},
{"id": "E-rev", "subject": "Bundle E review", "status": "completed", "blockedBy": ["E10", "E9"]},
{"id": "F11", "subject": "Bundle F T11 (controller-direct): Register ZB.MOM.WW.ScadaBridge.AuditLog in Component-Host.md + README confirm", "status": "completed", "blockedBy": ["E-rev"]},
{"id": "FINAL-rev", "subject": "Final cross-bundle review over the whole M1 branch", "status": "completed", "blockedBy": ["F11"]},
{"id": "MERGE", "subject": "Verify gate: full solution dotnet test green, then merge --no-ff to main", "status": "completed", "blockedBy": ["FINAL-rev"]},
{"id": "ROADMAP", "subject": "Update downstream M2-M8 sections of roadmap with realities learned in M1", "status": "completed", "blockedBy": ["MERGE"]}
],
"lastUpdated": "2026-05-20T00:00:00Z"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-20-centralized-audit-log.md.tasks.json
+20 -19
View File
@@ -3,24 +3,25 @@
"spec": "alog.md (commit fec0bb1)",
"repoNature": "design-documentation-only",
"tasks": [
{"id": 0, "subject": "Task 0: Prepare branch", "status": "pending", "blockedBy": []},
{"id": 1, "subject": "Task 1: Author Component-AuditLog.md", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Update Component-Commons.md", "status": "pending", "blockedBy": [0]},
{"id": 3, "subject": "Task 3: Update Component-ConfigurationDatabase.md", "status": "pending", "blockedBy": [1]},
{"id": 4, "subject": "Task 4: Update Component-ClusterInfrastructure.md", "status": "pending", "blockedBy": [1]},
{"id": 5, "subject": "Task 5: Update Component-SiteRuntime.md", "status": "pending", "blockedBy": [1]},
{"id": 6, "subject": "Task 6: Update Component-ExternalSystemGateway.md", "status": "pending", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: Update Component-SiteCallAudit.md", "status": "pending", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Update Component-NotificationOutbox.md", "status": "pending", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Update Component-InboundAPI.md", "status": "pending", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Update Component-CentralUI.md", "status": "pending", "blockedBy": [1]},
{"id": 11, "subject": "Task 11: Update Component-HealthMonitoring.md", "status": "pending", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: Update Component-CLI.md", "status": "pending", "blockedBy": [1]},
{"id": 13, "subject": "Task 13: Update README.md", "status": "pending", "blockedBy": [1]},
{"id": 14, "subject": "Task 14: Update HighLevelReqs.md", "status": "pending", "blockedBy": [1]},
{"id": 15, "subject": "Task 15: Update CLAUDE.md", "status": "pending", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 16: Final cross-reference verification", "status": "pending", "blockedBy": [15]},
{"id": 17, "subject": "Task 17: Merge to main (user-gated)", "status": "pending", "blockedBy": [16]}
{"id": 0, "subject": "Task 0: Prepare branch", "status": "completed", "blockedBy": []},
{"id": 1, "subject": "Task 1: Author Component-AuditLog.md", "status": "completed", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Update Component-Commons.md", "status": "completed", "blockedBy": [0]},
{"id": 3, "subject": "Task 3: Update Component-ConfigurationDatabase.md", "status": "completed", "blockedBy": [1]},
{"id": 4, "subject": "Task 4: Update Component-ClusterInfrastructure.md", "status": "completed", "blockedBy": [1]},
{"id": 5, "subject": "Task 5: Update Component-SiteRuntime.md", "status": "completed", "blockedBy": [1]},
{"id": 6, "subject": "Task 6: Update Component-ExternalSystemGateway.md", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "Task 7: Update Component-SiteCallAudit.md", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "Task 8: Update Component-NotificationOutbox.md", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Update Component-InboundAPI.md", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Update Component-CentralUI.md", "status": "completed", "blockedBy": [1]},
{"id": 11, "subject": "Task 11: Update Component-HealthMonitoring.md", "status": "completed", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: Update Component-CLI.md", "status": "completed", "blockedBy": [1]},
{"id": 13, "subject": "Task 13: Update README.md", "status": "completed", "blockedBy": [1]},
{"id": 14, "subject": "Task 14: Update HighLevelReqs.md", "status": "completed", "blockedBy": [1]},
{"id": 15, "subject": "Task 15: Update CLAUDE.md", "status": "completed", "blockedBy": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]},
{"id": 16, "subject": "Task 16: Final cross-reference verification", "status": "completed", "blockedBy": [15]},
{"id": 17, "subject": "Task 17: Merge to main (user-gated)", "status": "completed", "blockedBy": [16]}
],
"lastUpdated": "2026-05-20T00:00:00Z"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-21-audit-executionid.md.tasks.json
+12 -11
View File
@@ -1,16 +1,17 @@
{
"planPath": "docs/plans/2026-05-21-audit-executionid.md",
"tasks": [
{"id": 50, "subject": "Task 0: Prep — verify branch + baseline", "status": "pending"},
{"id": 51, "subject": "Task 1: Foundation — AuditEvent.ExecutionId + central AuditLog column + repo query", "status": "pending", "blockedBy": [50]},
{"id": 52, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "pending", "blockedBy": [51]},
{"id": 53, "subject": "Task 3: Site script-side emitters stamp ExecutionId", "status": "pending", "blockedBy": [52]},
{"id": 54, "subject": "Task 4: Cached S&F retry-loop rows carry ExecutionId", "status": "pending", "blockedBy": [53]},
{"id": 55, "subject": "Task 5: Central NotifyDeliver rows carry ExecutionId", "status": "pending", "blockedBy": [52]},
{"id": 56, "subject": "Task 6: Inbound audit rows carry ExecutionId", "status": "pending", "blockedBy": [52]},
{"id": 57, "subject": "Task 7: Central UI — ExecutionId column, filter, drill-in", "status": "pending", "blockedBy": [51]},
{"id": 58, "subject": "Task 8: CLI + ManagementService — ExecutionId filter", "status": "pending", "blockedBy": [51]},
{"id": 59, "subject": "Task 9: End-to-end integration test + docs", "status": "pending", "blockedBy": [53, 54, 55, 56, 57, 58]}
{"id": 50, "subject": "Task 0: Prep — verify branch + baseline", "status": "completed"},
{"id": 51, "subject": "Task 1: Foundation — AuditEvent.ExecutionId + central AuditLog column + repo query", "status": "completed", "blockedBy": [50]},
{"id": 52, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "completed", "blockedBy": [51]},
{"id": 53, "subject": "Task 3: Site script-side emitters stamp ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 54, "subject": "Task 4: Cached S&F retry-loop rows carry ExecutionId", "status": "completed", "blockedBy": [53]},
{"id": 55, "subject": "Task 5: Central NotifyDeliver rows carry ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 56, "subject": "Task 6: Inbound audit rows carry ExecutionId", "status": "completed", "blockedBy": [52]},
{"id": 57, "subject": "Task 7: Central UI — ExecutionId column, filter, drill-in", "status": "completed", "blockedBy": [51]},
{"id": 58, "subject": "Task 8: CLI + ManagementService — ExecutionId filter", "status": "completed", "blockedBy": [51]},
{"id": 59, "subject": "Task 9: End-to-end integration test + docs", "status": "completed", "blockedBy": [53, 54, 55, 56, 57, 58]}
],
"lastUpdated": "2026-05-21T00:00:00Z"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-21-audit-parent-executionid.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{
"planPath": "docs/plans/2026-05-21-audit-parent-executionid.md",
"tasks": [
{"id": 0, "subject": "Task 0: Prep — verify branch + baseline", "status": "pending"},
{"id": 1, "subject": "Task 1: Foundation — AuditEvent.ParentExecutionId + central AuditLog column", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Inbound request id minting + RouteToCallRequest.ParentExecutionId", "status": "pending", "blockedBy": [0]},
{"id": 4, "subject": "Task 4: Thread ParentExecutionId into routed script ScriptRuntimeContext", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Site script-side emitters stamp ParentExecutionId", "status": "pending", "blockedBy": [4, 2]},
{"id": 6, "subject": "Task 6: Cached S&F retry-loop rows carry ParentExecutionId", "status": "pending", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Central NotifyDeliver rows carry ParentExecutionId", "status": "pending", "blockedBy": [5, 1]},
{"id": 8, "subject": "Task 8: Repository — GetExecutionTreeAsync", "status": "pending", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Central UI — ParentExecutionId column, filter, parent drill-in", "status": "pending", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Central UI — execution-chain tree view", "status": "pending", "blockedBy": [8, 9]},
{"id": 11, "subject": "Task 11: CLI + ManagementService — ParentExecutionId filter", "status": "pending", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: End-to-end integration test + docs", "status": "pending", "blockedBy": [5, 6, 7, 10, 11]}
{"id": 0, "subject": "Task 0: Prep — verify branch + baseline", "status": "completed"},
{"id": 1, "subject": "Task 1: Foundation — AuditEvent.ParentExecutionId + central AuditLog column", "status": "completed", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: Foundation — site SQLite + gRPC DTO", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Inbound request id minting + RouteToCallRequest.ParentExecutionId", "status": "completed", "blockedBy": [0]},
{"id": 4, "subject": "Task 4: Thread ParentExecutionId into routed script ScriptRuntimeContext", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Site script-side emitters stamp ParentExecutionId", "status": "completed", "blockedBy": [4, 2]},
{"id": 6, "subject": "Task 6: Cached S&F retry-loop rows carry ParentExecutionId", "status": "completed", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Central NotifyDeliver rows carry ParentExecutionId", "status": "completed", "blockedBy": [5, 1]},
{"id": 8, "subject": "Task 8: Repository — GetExecutionTreeAsync", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "Task 9: Central UI — ParentExecutionId column, filter, parent drill-in", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "Task 10: Central UI — execution-chain tree view", "status": "completed", "blockedBy": [8, 9]},
{"id": 11, "subject": "Task 11: CLI + ManagementService — ParentExecutionId filter", "status": "completed", "blockedBy": [1]},
{"id": 12, "subject": "Task 12: End-to-end integration test + docs", "status": "completed", "blockedBy": [5, 6, 7, 10, 11]}
],
"lastUpdated": "2026-05-21"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-23-audit-source-node.md
+9 -9
View File
@@ -929,12 +929,12 @@ git commit -m "chore(audit): smoke-verify SourceNode end-to-end across cluster"
## Acceptance Criteria (the whole-plan checklist)
- [ ] Every audit row written from this commit forward carries `SourceNode` populated (`central-a/b` for central direct-write, `node-a/b` for site rows).
- [ ] Every new `Notifications` and `SiteCalls` row carries `SourceNode` from the site.
- [ ] `IX_AuditLog_Node_Occurred` exists in central MS SQL.
- [ ] Site SQLite `AuditLog` and `OperationTracking` tables both have `SourceNode TEXT NULL`; existing site DBs are upgraded idempotently on startup.
- [ ] Proto `AuditEventDto.source_node = 22` and `SiteCallOperationalDto.source_node = 12` exist; no field numbers reused.
- [ ] Central UI Audit Log, Notifications, and Site Calls pages all display a "Node" column and support filtering by it.
- [ ] All test projects green.
- [ ] Cluster comes up clean via `bash docker/deploy.sh`; CLI smoke confirms expected node names land in the central tables.
- [ ] Design docs (already committed in Task 0) match the implementation.
- [x] Every audit row written from this commit forward carries `SourceNode` populated (`central-a/b` for central direct-write, `node-a/b` for site rows).
- [x] Every new `Notifications` and `SiteCalls` row carries `SourceNode` from the site.
- [x] `IX_AuditLog_Node_Occurred` exists in central MS SQL.
- [x] Site SQLite `AuditLog` and `OperationTracking` tables both have `SourceNode TEXT NULL`; existing site DBs are upgraded idempotently on startup.
- [x] Proto `AuditEventDto.source_node = 22` and `SiteCallOperationalDto.source_node = 12` exist; no field numbers reused.
- [x] Central UI Audit Log, Notifications, and Site Calls pages all display a "Node" column and support filtering by it.
- [x] All test projects green.
- [x] Cluster comes up clean via `bash docker/deploy.sh`; CLI smoke confirms expected node names land in the central tables. (`NodeName` is now startup-validated — an empty value fails fast rather than NULLing `SourceNode`, commit `4a0462e4` — and the rig has redeployed many times since.)
- [x] Design docs (already committed in Task 0) match the implementation.
docs/plans/2026-05-23-audit-source-node.md.tasks.json
+23 -22
View File
@@ -1,27 +1,28 @@
{
"planPath": "docs/plans/2026-05-23-audit-source-node.md",
"tasks": [
{"id": 1, "subject": "Task 0: Branch + Snapshot", "status": "pending"},
{"id": 2, "subject": "Task 1: NodeOptions.NodeName + INodeIdentityProvider", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Add SourceNode to AuditEvent record", "status": "pending", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Add SourceNode to SiteCallOperational + SiteCall entity", "status": "pending", "blockedBy": [2]},
{"id": 5, "subject": "Task 4: Add SourceNode to Notification entity + NotificationSubmit", "status": "pending", "blockedBy": [2]},
{"id": 6, "subject": "Task 5: Add source_node to proto + update DTO mappers", "status": "pending", "blockedBy": [3, 4]},
{"id": 7, "subject": "Task 6: EF migration — SourceNode on AuditLog + IX_AuditLog_Node_Occurred", "status": "pending", "blockedBy": [3]},
{"id": 8, "subject": "Task 7: EF migration — SourceNode on Notifications", "status": "pending", "blockedBy": [5]},
{"id": 9, "subject": "Task 8: EF migration — SourceNode on SiteCalls", "status": "pending", "blockedBy": [4]},
{"id": 10, "subject": "Task 9: Site SQLite AuditLog — add SourceNode (idempotent upgrade)", "status": "pending", "blockedBy": [3]},
{"id": 11, "subject": "Task 10: Site SQLite OperationTracking — add SourceNode", "status": "pending", "blockedBy": [4]},
{"id": 12, "subject": "Task 11: Stamp SourceNode at site SqliteAuditWriter", "status": "pending", "blockedBy": [2, 10]},
{"id": 13, "subject": "Task 12: Stamp SourceNode at CentralAuditWriter + persist via repo", "status": "pending", "blockedBy": [2, 7]},
{"id": 14, "subject": "Task 13: Carry SourceNode through Notifications S&F handoff", "status": "pending", "blockedBy": [2, 5, 8]},
{"id": 15, "subject": "Task 14: Carry SourceNode through cached-call telemetry → SiteCalls", "status": "pending", "blockedBy": [2, 9, 11]},
{"id": 16, "subject": "Task 15: UI — Node column + filter on AuditLog grid", "status": "pending", "blockedBy": [7]},
{"id": 17, "subject": "Task 16: UI — Node column + filter on Notifications grid", "status": "pending", "blockedBy": [7]},
{"id": 18, "subject": "Task 17: UI — Node column + filter on SiteCalls grid", "status": "pending", "blockedBy": [7]},
{"id": 19, "subject": "Task 18: Docker appsettings — NodeName on all 8 nodes", "status": "pending", "blockedBy": [2]},
{"id": 20, "subject": "Task 19: Full build + targeted test sweep", "status": "pending", "blockedBy": [12, 13, 14, 15, 16, 17, 18, 19]},
{"id": 21, "subject": "Task 20: Docker redeploy + smoke verify", "status": "pending", "blockedBy": [20]}
{"id": 1, "subject": "Task 0: Branch + Snapshot", "status": "completed"},
{"id": 2, "subject": "Task 1: NodeOptions.NodeName + INodeIdentityProvider", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Add SourceNode to AuditEvent record", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Add SourceNode to SiteCallOperational + SiteCall entity", "status": "completed", "blockedBy": [2]},
{"id": 5, "subject": "Task 4: Add SourceNode to Notification entity + NotificationSubmit", "status": "completed", "blockedBy": [2]},
{"id": 6, "subject": "Task 5: Add source_node to proto + update DTO mappers", "status": "completed", "blockedBy": [3, 4]},
{"id": 7, "subject": "Task 6: EF migration — SourceNode on AuditLog + IX_AuditLog_Node_Occurred", "status": "completed", "blockedBy": [3]},
{"id": 8, "subject": "Task 7: EF migration — SourceNode on Notifications", "status": "completed", "blockedBy": [5]},
{"id": 9, "subject": "Task 8: EF migration — SourceNode on SiteCalls", "status": "completed", "blockedBy": [4]},
{"id": 10, "subject": "Task 9: Site SQLite AuditLog — add SourceNode (idempotent upgrade)", "status": "completed", "blockedBy": [3]},
{"id": 11, "subject": "Task 10: Site SQLite OperationTracking — add SourceNode", "status": "completed", "blockedBy": [4]},
{"id": 12, "subject": "Task 11: Stamp SourceNode at site SqliteAuditWriter", "status": "completed", "blockedBy": [2, 10]},
{"id": 13, "subject": "Task 12: Stamp SourceNode at CentralAuditWriter + persist via repo", "status": "completed", "blockedBy": [2, 7]},
{"id": 14, "subject": "Task 13: Carry SourceNode through Notifications S&F handoff", "status": "completed", "blockedBy": [2, 5, 8]},
{"id": 15, "subject": "Task 14: Carry SourceNode through cached-call telemetry → SiteCalls", "status": "completed", "blockedBy": [2, 9, 11]},
{"id": 16, "subject": "Task 15: UI — Node column + filter on AuditLog grid", "status": "completed", "blockedBy": [7]},
{"id": 17, "subject": "Task 16: UI — Node column + filter on Notifications grid", "status": "completed", "blockedBy": [7]},
{"id": 18, "subject": "Task 17: UI — Node column + filter on SiteCalls grid", "status": "completed", "blockedBy": [7]},
{"id": 19, "subject": "Task 18: Docker appsettings — NodeName on all 8 nodes", "status": "completed", "blockedBy": [2]},
{"id": 20, "subject": "Task 19: Full build + targeted test sweep", "status": "completed", "blockedBy": [12, 13, 14, 15, 16, 17, 18, 19]},
{"id": 21, "subject": "Task 20: Docker redeploy + smoke verify", "status": "completed", "blockedBy": [20]}
],
"lastUpdated": "2026-05-23T00:00:00Z"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-23-inbound-api-full-response-audit-design.md
+7 -7
View File
@@ -132,17 +132,17 @@ same "options validation" path used for other AuditLog settings.
## Acceptance Criteria
- [ ] `AuditLog:InboundMaxBytes` option exists on the AuditLog options class,
- [x] `AuditLog:InboundMaxBytes` option exists on the AuditLog options class,
with the documented default and bounds, validated at startup.
- [ ] Inbound request middleware writes `RequestSummary` and `ResponseSummary`
- [x] Inbound request middleware writes `RequestSummary` and `ResponseSummary`
using the inbound ceiling instead of the 8 KB / 64 KB defaults.
- [ ] Other channels' rows (e.g. an `ApiOutbound.ApiCall` over the limit) still
- [x] Other channels' rows (e.g. an `ApiOutbound.ApiCall` over the limit) still
truncate at 8 KB (64 KB on error rows) — regression-tested.
- [ ] `PayloadTruncated = 1` on an inbound row iff request body or response
- [x] `PayloadTruncated = 1` on an inbound row iff request body or response
body exceeded `InboundMaxBytes`.
- [ ] Header redaction list and per-target body redactors still apply to
- [x] Header redaction list and per-target body redactors still apply to
inbound rows.
- [ ] Redactor failure on an inbound row still produces `<redacted: redactor
- [x] Redactor failure on an inbound row still produces `<redacted: redactor
error>` and increments `AuditRedactionFailure`.
- [ ] `Component-AuditLog.md` and `Component-InboundAPI.md` updated as
- [x] `Component-AuditLog.md` and `Component-InboundAPI.md` updated as
described in **Doc Edits**.
docs/plans/2026-05-23-inbound-api-full-response-audit.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{
"planPath": "docs/plans/2026-05-23-inbound-api-full-response-audit.md",
"tasks": [
{"id": 1, "subject": "Task 0: Prep — branch, baseline build", "status": "pending"},
{"id": 2, "subject": "Task 1: Add InboundMaxBytes to AuditLogOptions (TDD)", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Wire InboundMaxBytes into DefaultAuditPayloadFilter (TDD)", "status": "pending", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Capture response body in AuditWriteMiddleware (TDD)", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "Task 4: Update Component-AuditLog.md", "status": "pending", "blockedBy": [4]},
{"id": 6, "subject": "Task 5: Update Component-InboundAPI.md", "status": "pending", "blockedBy": [5]},
{"id": 7, "subject": "Task 6: Final build + full test run + branch summary", "status": "pending", "blockedBy": [6]}
{"id": 1, "subject": "Task 0: Prep — branch, baseline build", "status": "completed"},
{"id": 2, "subject": "Task 1: Add InboundMaxBytes to AuditLogOptions (TDD)", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 2: Wire InboundMaxBytes into DefaultAuditPayloadFilter (TDD)", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 3: Capture response body in AuditWriteMiddleware (TDD)", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 4: Update Component-AuditLog.md", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "Task 5: Update Component-InboundAPI.md", "status": "completed", "blockedBy": [5]},
{"id": 7, "subject": "Task 6: Final build + full test run + branch summary", "status": "completed", "blockedBy": [6]}
],
"lastUpdated": "2026-05-23"
"lastUpdated": "2026-08-01",
"note": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-05-24-second-environment-verification.md
+16
View File
@@ -1,5 +1,21 @@
# Env2 + Transport Manual Verification Checklist
> **RETIRED 2026-08-01 — never run as written; superseded by automated suites + a live cross-environment run.**
> The step list below predates the M8 import-wizard Map step and was never annotated. Instead of
> refreshing ~48 stale steps, the core scenario this checklist exists for was run live on 2026-08-01
> (rig session, task #12): CLI `bundle export` from the primary cluster (encrypted, `--include-dependencies`,
> template closure + shared scripts + external system + site-a + instances) → `bundle preview` against the
> empty env2 (correct Map requirements surfaced) → `bundle import --create-missing-sites
> --create-missing-connections` into env2 → 20 entities added, secrets (`AuthConfiguration`) intact,
> instances landed `NotDeployed`, connection FK wired to the created site. **The run caught a real bug**
> (create-missing site id never materialised before the connection insert → FK 547 on real SQL Server,
> masked by the in-memory test provider) — fixed the same day with an SQLite regression test
> (`CreateMissingSiteRelationalTests`, commit `0c9dffed`). Env2 boot/PSK/LocalDb were separately proven
> PASS 3/3 by Gitea #31 (2026-07-23). Remaining coverage lives in
> `tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests` (RoundTrip/RoundTripEquivalence/
> ConflictResolution/ValidationFailure/SiteInstanceImport/CreateMissingSiteRelational). This checklist
> is retired — do not run or refresh it.
**Date created:** 2026-05-24
**Companion to:** [`2026-05-24-second-environment-design.md`](2026-05-24-second-environment-design.md), [`Component-Transport.md`](../requirements/Component-Transport.md)
**Goal:** Exercise the Transport (#24) bundle export/import flow against two real running environments (primary + env2).
docs/plans/2026-05-24-transport-manual-verification.md
+7
View File
@@ -1,5 +1,12 @@
# Transport Manual Verification
> **RETIRED 2026-08-01 — never run as written; ~80% overlap with
> `2026-05-24-second-environment-verification.md`, which carries the full retirement rationale.**
> The steps predate the M8 import-wizard Map step (numbering stale). The export→preview→import flow
> was proven live cross-environment on 2026-08-01 via the CLI (see the retirement banner in the env2
> checklist), and the UI wizard surface is covered by the Transport Playwright/bUnit suites. Retired —
> do not run or refresh.
This document is a one-time manual verification to be run against the docker cluster after the Transport feature is fully built and `bash docker/deploy.sh` has rebuilt the image. Each step is sequential and assumes the previous step succeeded. The entire flow takes approximately 15 minutes.
## Prerequisites
docs/plans/2026-05-28-opcua-tag-browser.md
+10
View File
@@ -1758,6 +1758,16 @@ git commit -m "feat(centralui): add OPC UA browse button + override column to In
### Task 19: End-to-end smoke (manual)
> **PASS 2026-08-01 (online + offline)** — run live on the docker rig against a purpose-built
> deployed instance (data-sourced Float attribute bound to site-a's "OPC PLC Simulator";
> artifacts deleted after). Online: Browse dialog opened, root populated (Server visible),
> expanded OpcPlc → Telemetry → Basic, selected `StepUp` — footer + override stored the durable
> `nsu=http://microsoft.com/Opc/OpcPlc/;s=StepUp` form and it persisted across save + reload.
> Offline: with BOTH site-a nodes stopped, Browse showed the error banner
> (`StatusCode="Unavailable"` + Retry) with the manual-paste field still usable; a hand-typed
> `ns=3;s=TestChildObject.TestInt` (bare-index advisory shown as designed) went Use → Select →
> Save and persisted across reload while the site was down. Pair restarted staggered and rejoined.
**Classification:** trivial
**Estimated implement time:** ~5 min (manual)
**Parallelizable with:** none (validates Tasks 1–18 together)
docs/plans/2026-05-28-opcua-tag-browser.md.tasks.json
+24 -23
View File
@@ -1,28 +1,29 @@
{
"planPath": "docs/plans/2026-05-28-opcua-tag-browser.md",
"tasks": [
{"id": 70, "subject": "Task 1: Add DataSourceReferenceOverride to InstanceConnectionBinding entity", "status": "pending"},
{"id": 71, "subject": "Task 2: Add override to ConnectionBinding wire record + ManagementActor mapping", "status": "pending"},
{"id": 72, "subject": "Task 3: EF mapping for DataSourceReferenceOverride column", "status": "pending"},
{"id": 73, "subject": "Task 4: EF Core migration AddInstanceConnectionBindingOverride", "status": "pending", "blockedBy": [72]},
{"id": 74, "subject": "Task 5: IBrowsableDataConnection interface + BrowseNode types", "status": "pending"},
{"id": 75, "subject": "Task 6: BrowseCommands.cs (BrowseOpcUaNodeCommand + result + failure)", "status": "pending"},
{"id": 76, "subject": "Task 7: Add BrowseChildrenAsync to IOpcUaClient", "status": "pending", "blockedBy": [74, 75]},
{"id": 77, "subject": "Task 8: Implement BrowseChildrenAsync on RealOpcUaClient", "status": "pending", "blockedBy": [76]},
{"id": 78, "subject": "Task 9: Implement IBrowsableDataConnection on OpcUaDataConnection", "status": "pending", "blockedBy": [76]},
{"id": 79, "subject": "Task 10: Handle BrowseOpcUaNodeCommand in DataConnectionManagerActor", "status": "pending", "blockedBy": [75, 76]},
{"id": 80, "subject": "Task 11: Forward BrowseOpcUaNodeCommand in SiteCommunicationActor", "status": "pending", "blockedBy": [79]},
{"id": 81, "subject": "Task 12: Apply override in FlatteningService.ApplyConnectionBindings", "status": "pending", "blockedBy": [70]},
{"id": 82, "subject": "Task 13: Revision-hash regression test", "status": "pending", "blockedBy": [81]},
{"id": 83, "subject": "Task 14: IOpcUaBrowseService + impl + DI registration", "status": "pending", "blockedBy": [75]},
{"id": 84, "subject": "Task 15: Scaffold OpcUaBrowserDialog modal", "status": "pending"},
{"id": 85, "subject": "Task 16: Tree rendering + lazy load + selection in dialog", "status": "pending", "blockedBy": [83, 84]},
{"id": 86, "subject": "Task 17: Error banner mapping + polish on dialog", "status": "pending", "blockedBy": [85]},
{"id": 87, "subject": "Task 18: Add Override column + Browse button to InstanceConfigure.razor", "status": "pending", "blockedBy": [83, 86]},
{"id": 88, "subject": "Task 19: End-to-end manual smoke (online + offline)", "status": "pending", "blockedBy": [70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87]},
{"id": 89, "subject": "Task 20: Update Component-DataConnectionLayer.md", "status": "pending", "blockedBy": [81]},
{"id": 90, "subject": "Task 21: Update Component-TemplateEngine.md", "status": "pending", "blockedBy": [81]},
{"id": 91, "subject": "Task 22: Update Component-CentralUI.md", "status": "pending", "blockedBy": [87]}
{"id": 70, "subject": "Task 1: Add DataSourceReferenceOverride to InstanceConnectionBinding entity", "status": "completed"},
{"id": 71, "subject": "Task 2: Add override to ConnectionBinding wire record + ManagementActor mapping", "status": "completed"},
{"id": 72, "subject": "Task 3: EF mapping for DataSourceReferenceOverride column", "status": "completed"},
{"id": 73, "subject": "Task 4: EF Core migration AddInstanceConnectionBindingOverride", "status": "completed", "blockedBy": [72]},
{"id": 74, "subject": "Task 5: IBrowsableDataConnection interface + BrowseNode types", "status": "completed"},
{"id": 75, "subject": "Task 6: BrowseCommands.cs (BrowseOpcUaNodeCommand + result + failure)", "status": "completed"},
{"id": 76, "subject": "Task 7: Add BrowseChildrenAsync to IOpcUaClient", "status": "completed", "blockedBy": [74, 75]},
{"id": 77, "subject": "Task 8: Implement BrowseChildrenAsync on RealOpcUaClient", "status": "completed", "blockedBy": [76]},
{"id": 78, "subject": "Task 9: Implement IBrowsableDataConnection on OpcUaDataConnection", "status": "completed", "blockedBy": [76]},
{"id": 79, "subject": "Task 10: Handle BrowseOpcUaNodeCommand in DataConnectionManagerActor", "status": "completed", "blockedBy": [75, 76]},
{"id": 80, "subject": "Task 11: Forward BrowseOpcUaNodeCommand in SiteCommunicationActor", "status": "completed", "blockedBy": [79]},
{"id": 81, "subject": "Task 12: Apply override in FlatteningService.ApplyConnectionBindings", "status": "completed", "blockedBy": [70]},
{"id": 82, "subject": "Task 13: Revision-hash regression test", "status": "completed", "blockedBy": [81]},
{"id": 83, "subject": "Task 14: IOpcUaBrowseService + impl + DI registration", "status": "completed", "blockedBy": [75]},
{"id": 84, "subject": "Task 15: Scaffold OpcUaBrowserDialog modal", "status": "completed"},
{"id": 85, "subject": "Task 16: Tree rendering + lazy load + selection in dialog", "status": "completed", "blockedBy": [83, 84]},
{"id": 86, "subject": "Task 17: Error banner mapping + polish on dialog", "status": "completed", "blockedBy": [85]},
{"id": 87, "subject": "Task 18: Add Override column + Browse button to InstanceConfigure.razor", "status": "completed", "blockedBy": [83, 86]},
{"id": 88, "subject": "Task 19: End-to-end manual smoke (online + offline)", "status": "completed", "blockedBy": [70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87], "notes": "Completed 2026-08-01: manual smoke PASS, online + offline paths verified live (commit 6dc5d94c). Status flipped from stale 'pending' by the 2026-08-07 truth sweep."},
{"id": 89, "subject": "Task 20: Update Component-DataConnectionLayer.md", "status": "completed", "blockedBy": [81]},
{"id": 90, "subject": "Task 21: Update Component-TemplateEngine.md", "status": "completed", "blockedBy": [81]},
{"id": 91, "subject": "Task 22: Update Component-CentralUI.md", "status": "completed", "blockedBy": [87]}
],
"lastUpdated": "2026-05-28T00:00:00Z"
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"lastUpdated": "2026-08-01"
}
docs/plans/2026-05-28-scadabridge-rename-design.md
+2 -2
View File
@@ -1,7 +1,7 @@
# ScadaLink → ZB.MOM.WW.ScadaBridge Rename — Design
**Date:** 2026-05-28
**Status:** Approved, in implementation
**Status:** Implemented (commit `7b0b9c73`)
**Scope:** Repo-wide rename of the product from "ScadaLink" to "ScadaBridge" and addition of the `ZB.MOM.WW` company prefix to every .NET project. Code, runtime artifacts (containers, network, databases), docs, and CLI config.
## Decisions
@@ -115,7 +115,7 @@ Drop volume + re-seed. `docker compose -f infra/docker-compose.yml down -v` then
## Out of Scope
- Sister repos `~/Desktop/MxAccessGateway`, `~/Desktop/OtOpcUa` — independent codebases, independent timelines.
- Repo folder `~/Desktop/scadalink-design` — left as-is to preserve Claude Code memory paths and user shell context.
- Repo folder `~/Desktop/scadalink-design` — left as-is to preserve Claude Code memory paths and user shell context. *(2026-08-01: this exclusion was later reversed and completed — see `2026-05-31-folder-repo-rename-scadabridge-plan.md` / `-design.md`, which renamed the folder and the Gitea repo to ScadaBridge.)*
- Existing Transport bundle manifests with `SourceEnvironment = "docker-cluster"` — those values are deployment IDs and stay stable. If the user later wants to rename the deployment IDs themselves, that's a separate design.
## Verification
docs/plans/2026-06-03-component-reference-docs.md.tasks.json
+30 -29
View File
@@ -1,34 +1,35 @@
{
"planPath": "docs/plans/2026-06-03-component-reference-docs.md",
"tasks": [
{"id": 19, "subject": "Task 0: Scaffold + shared assets", "status": "pending"},
{"id": 20, "subject": "Task 1: Pilot exemplar AuditLog.md (approval gate)", "status": "pending", "blockedBy": [19]},
{"id": 21, "subject": "Task 2: Commons.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 22, "subject": "Task 3: ConfigurationDatabase.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 23, "subject": "Task 4: Communication.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 24, "subject": "Task 5: ClusterInfrastructure.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 25, "subject": "Task 6: Host.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 26, "subject": "Task 7: Security.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 27, "subject": "Task 8: TemplateEngine.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 28, "subject": "Task 9: DeploymentManager.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 29, "subject": "Task 10: SiteRuntime.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 30, "subject": "Task 11: DataConnectionLayer.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 31, "subject": "Task 12: StoreAndForward.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 32, "subject": "Task 13: ExternalSystemGateway.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 33, "subject": "Task 14: NotificationService.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 34, "subject": "Task 15: NotificationOutbox.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 35, "subject": "Task 16: SiteCallAudit.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 36, "subject": "Task 17: HealthMonitoring.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 37, "subject": "Task 18: SiteEventLogging.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 38, "subject": "Task 19: InboundAPI.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 39, "subject": "Task 20: ManagementService.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 40, "subject": "Task 21: CLI.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 41, "subject": "Task 22: Transport.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 42, "subject": "Task 23: CentralUI.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 43, "subject": "Task 24: TraefikProxy.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 44, "subject": "Task 25: TreeView.md reference doc", "status": "pending", "blockedBy": [20]},
{"id": 45, "subject": "Task 26: Index + README link", "status": "pending", "blockedBy": [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]},
{"id": 46, "subject": "Task 27: Verification & fix pass", "status": "pending", "blockedBy": [45]}
{"id": 19, "subject": "Task 0: Scaffold + shared assets", "status": "completed"},
{"id": 20, "subject": "Task 1: Pilot exemplar AuditLog.md (approval gate)", "status": "completed", "blockedBy": [19]},
{"id": 21, "subject": "Task 2: Commons.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 22, "subject": "Task 3: ConfigurationDatabase.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 23, "subject": "Task 4: Communication.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 24, "subject": "Task 5: ClusterInfrastructure.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 25, "subject": "Task 6: Host.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 26, "subject": "Task 7: Security.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 27, "subject": "Task 8: TemplateEngine.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 28, "subject": "Task 9: DeploymentManager.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 29, "subject": "Task 10: SiteRuntime.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 30, "subject": "Task 11: DataConnectionLayer.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 31, "subject": "Task 12: StoreAndForward.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 32, "subject": "Task 13: ExternalSystemGateway.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 33, "subject": "Task 14: NotificationService.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 34, "subject": "Task 15: NotificationOutbox.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 35, "subject": "Task 16: SiteCallAudit.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 36, "subject": "Task 17: HealthMonitoring.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 37, "subject": "Task 18: SiteEventLogging.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 38, "subject": "Task 19: InboundAPI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 39, "subject": "Task 20: ManagementService.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 40, "subject": "Task 21: CLI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 41, "subject": "Task 22: Transport.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 42, "subject": "Task 23: CentralUI.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 43, "subject": "Task 24: TraefikProxy.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 44, "subject": "Task 25: TreeView.md reference doc", "status": "completed", "blockedBy": [20]},
{"id": 45, "subject": "Task 26: Index + README link", "status": "completed", "blockedBy": [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]},
{"id": 46, "subject": "Task 27: Verification & fix pass", "status": "completed", "blockedBy": [45]}
],
"lastUpdated": "2026-06-03"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-05-playwright-coverage-expansion.md.tasks.json
+18 -17
View File
@@ -1,23 +1,24 @@
{
"planPath": "docs/plans/2026-06-05-playwright-coverage-expansion.md",
"lastUpdated": "2026-06-05T00:00:00Z",
"lastUpdated": "2026-08-01T00:00:00Z",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code",
"nativeTaskIdBase": 57,
"tasks": [
{"id": 0, "nativeId": 57, "subject": "Task 0: Add CLI ProjectReference to test project", "status": "pending"},
{"id": 1, "nativeId": 58, "subject": "Task 1: CliRunner core + ClusterAvailability probe", "status": "pending", "blockedBy": [0]},
{"id": 2, "nativeId": 59, "subject": "Task 2: CliRunner typed fixture helpers", "status": "pending", "blockedBy": [1]},
{"id": 3, "nativeId": 60, "subject": "Task 3: Standardize skip policy + skip-count logging", "status": "pending", "blockedBy": [1]},
{"id": 4, "nativeId": 61, "subject": "Task 4: DeploymentFixture (ephemeral instance on site-a)", "status": "pending", "blockedBy": [2]},
{"id": 5, "nativeId": 62, "subject": "Task 5: DeploymentActionTests.Deploy", "status": "pending", "blockedBy": [4]},
{"id": 6, "nativeId": 63, "subject": "Task 6: DeploymentActionTests.Enable + Disable", "status": "pending", "blockedBy": [5]},
{"id": 7, "nativeId": 64, "subject": "Task 7: DeploymentActionTests.Delete", "status": "pending", "blockedBy": [6]},
{"id": 8, "nativeId": 65, "subject": "Task 8: Notification retry/discard + ParkedMessages query", "status": "pending", "blockedBy": [2]},
{"id": 9, "nativeId": 66, "subject": "Task 9: Transport Import round-trip", "status": "pending", "blockedBy": [2]},
{"id": 10, "nativeId": 67, "subject": "Task 10: Site CRUD round-trip", "status": "pending", "blockedBy": [2]},
{"id": 11, "nativeId": 68, "subject": "Task 11: Template CRUD round-trip", "status": "pending", "blockedBy": [2]},
{"id": 12, "nativeId": 69, "subject": "Task 12: LDAP mapping CRUD round-trip", "status": "pending", "blockedBy": [1]},
{"id": 13, "nativeId": 70, "subject": "Task 13: Navigation render-assertion hardening", "status": "pending"},
{"id": 14, "nativeId": 71, "subject": "Task 14: Health KPI load test", "status": "pending", "blockedBy": [1]},
{"id": 15, "nativeId": 72, "subject": "Task 15: Full-suite verification + no-residue check", "status": "pending", "blockedBy": [3, 7, 8, 9, 10, 11, 12, 13, 14]}
{"id": 0, "nativeId": 57, "subject": "Task 0: Add CLI ProjectReference to test project", "status": "completed"},
{"id": 1, "nativeId": 58, "subject": "Task 1: CliRunner core + ClusterAvailability probe", "status": "completed", "blockedBy": [0]},
{"id": 2, "nativeId": 59, "subject": "Task 2: CliRunner typed fixture helpers", "status": "completed", "blockedBy": [1]},
{"id": 3, "nativeId": 60, "subject": "Task 3: Standardize skip policy + skip-count logging", "status": "completed", "blockedBy": [1]},
{"id": 4, "nativeId": 61, "subject": "Task 4: DeploymentFixture (ephemeral instance on site-a)", "status": "completed", "blockedBy": [2]},
{"id": 5, "nativeId": 62, "subject": "Task 5: DeploymentActionTests.Deploy", "status": "completed", "blockedBy": [4]},
{"id": 6, "nativeId": 63, "subject": "Task 6: DeploymentActionTests.Enable + Disable", "status": "completed", "blockedBy": [5]},
{"id": 7, "nativeId": 64, "subject": "Task 7: DeploymentActionTests.Delete", "status": "completed", "blockedBy": [6]},
{"id": 8, "nativeId": 65, "subject": "Task 8: Notification retry/discard + ParkedMessages query", "status": "completed", "blockedBy": [2]},
{"id": 9, "nativeId": 66, "subject": "Task 9: Transport Import round-trip", "status": "completed", "blockedBy": [2]},
{"id": 10, "nativeId": 67, "subject": "Task 10: Site CRUD round-trip", "status": "completed", "blockedBy": [2]},
{"id": 11, "nativeId": 68, "subject": "Task 11: Template CRUD round-trip", "status": "completed", "blockedBy": [2]},
{"id": 12, "nativeId": 69, "subject": "Task 12: LDAP mapping CRUD round-trip", "status": "completed", "blockedBy": [1]},
{"id": 13, "nativeId": 70, "subject": "Task 13: Navigation render-assertion hardening", "status": "completed"},
{"id": 14, "nativeId": 71, "subject": "Task 14: Health KPI load test", "status": "completed", "blockedBy": [1]},
{"id": 15, "nativeId": 72, "subject": "Task 15: Full-suite verification + no-residue check", "status": "completed", "blockedBy": [3, 7, 8, 9, 10, 11, 12, 13, 14]}
]
}
docs/plans/2026-06-15-stillpending-completion-design.md
+5 -2
View File
@@ -78,7 +78,10 @@ Wire up behavior that exists in code but is never started, and fill the event-lo
#### M5 — Audit hardening (T1–T8)
Hash-chain tamper evidence (off by default, `verify-chain` made real); Parquet export/archival (replace the 501); per-channel retention overrides; tag-cascade for `ParentExecutionId` (thread writing-execution id through trigger-driven runs); ExecutionId/ParentExecutionId + SourceNode backfill on historical rows; per-node stuck-count KPIs; structured response capture (headers/content-type, inbound request headers, per-method opt-out, `AuditInboundCeilingHits` metric); CLI `audit tree`.
#### M6 — KPI History & Trends (T11 delivered; T9/T10 deferred)
#### M6 — KPI History & Trends (T11 delivered; T9/T10 **since DELIVERED 2026-06-19**)
> **2026-08-01 bookkeeping sync:** the T9/T10 "deferred to the next major version" text below is superseded — **T9/T10 were delivered 2026-06-19 as the SMS (Twilio) adapter** (`SmsNotificationDeliveryAdapter`, `NotificationType.Sms`, Central UI Type selector + SMS recipient input; see `docs/plans/2026-06-19-sms-notifications.md`). **Teams was evaluated and dropped by design decision** — outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only.
Reshaped during the 2026-06-17 brainstorm (see `docs/plans/2026-06-17-m6-kpi-history-design.md`):
- **T11 — DELIVERED** as the reusable **KPI-history backbone** (#26 KpiHistory), promoted from a notifications-only feature. A tall/EAV `KpiSample` store in **central MS SQL** (no new infra — supersedes the original "point-in-time only, no time-series store" stance), a `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated, best-effort with per-source isolation) sampling DI-registered `IKpiSampleSource`s every minute, a bucketed `GetRawSeriesAsync` + `KpiSeriesBucketer` query + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart` (no third-party charting lib). Trends shipped for **all** current KPI sources — Notification Outbox, Site Call Audit, Audit Log, and Site Health — across four UI surfaces.
- **T9 (Teams + other non-Email delivery adapters behind `INotificationDeliveryAdapter`) — DEFERRED to the next major version.** The seam exists; no code now. Transport choice (Incoming Webhook vs Microsoft Graph) and the Teams list-targeting model remain to be designed.
@@ -132,7 +135,7 @@ This is the **final milestone** of the system-completion roadmap. All in-scope M
## Dependencies & sequencing
- **M1 → M5** — audit hardening builds on the wired purge/reconciliation.
- **M6/T11** — delivered as the #26 KpiHistory backbone; reused **central MS SQL** (a tall/EAV `KpiSample` table) rather than introducing new infra. T9/T10 deferred to the next major version.
- **M6/T11** — delivered as the #26 KpiHistory backbone; reused **central MS SQL** (a tall/EAV `KpiSample` table) rather than introducing new infra. T9/T10 since delivered 2026-06-19 as the SMS (Twilio) adapter (see note above).
- **M9/T26** — base-template versioning is the largest authoring item; may split.
- **M4** — runs anytime; cheap and high-clarity, good to interleave.
- **M3** — independent; can run in parallel with M1/M2.
docs/plans/2026-06-15-stillpending-m2-implementation.md.tasks.json
+5 -4
View File
@@ -27,9 +27,10 @@
{"ref": "#19", "subject": "script started/completed events", "status": "done in M1.8"}
],
"followups": [
{"id": 52, "subject": "Investigate 2 partition-purge E2E test failures (AuditLogPurgeActor/PartitionPurge)", "from": "M2.0", "status": "pending"},
{"id": 53, "subject": "Dedup alarm-capable protocol predicate (3 copies → AlarmCapableProtocols)", "from": "M2.1", "status": "pending"},
{"id": 54, "subject": "Expose ExecutionTimeoutSeconds (+ MinTimeBetweenRuns) in CLI + UI script authoring", "from": "M2.5", "status": "pending"}
{"id": 52, "subject": "Investigate 2 partition-purge E2E test failures (AuditLogPurgeActor/PartitionPurge)", "from": "M2.0", "status": "resolved", "note": "RESOLVED — commit 639e331d de-dated the 2 EndToEnd purge tests (M5.7)"},
{"id": 53, "subject": "Dedup alarm-capable protocol predicate (3 copies → AlarmCapableProtocols)", "from": "M2.1", "status": "resolved", "note": "RESOLVED — shared Commons/Interfaces/Protocol/AlarmCapableProtocols.cs is now the single predicate"},
{"id": 54, "subject": "Expose ExecutionTimeoutSeconds (+ MinTimeBetweenRuns) in CLI + UI script authoring", "from": "M2.5", "status": "resolved", "note": "RESOLVED — CLI --execution-timeout-seconds on template script add/update plus TemplateEdit binding"}
],
"lastUpdated": "2026-06-15"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-15-stillpending-phase1-implementation.md.tasks.json
+15 -14
View File
@@ -1,19 +1,20 @@
{
"planPath": "docs/plans/2026-06-15-stillpending-phase1-implementation.md",
"tasks": [
{"id": 22, "subject": "M1.0: Confirm proto/site surface for audit pull (spike)", "status": "pending"},
{"id": 23, "subject": "M1.1: Production gRPC IPullAuditEventsClient", "status": "pending", "blockedBy": [22]},
{"id": 24, "subject": "M1.2: Wire reconciliation + purge actors as central singletons", "status": "pending"},
{"id": 25, "subject": "M1.3: SiteCallAudit periodic reconciliation pull", "status": "pending"},
{"id": 26, "subject": "M1.4: SiteCallAudit daily terminal-row purge scheduler", "status": "pending"},
{"id": 27, "subject": "M1.5: SiteEventLog — emit Alarm events", "status": "pending"},
{"id": 28, "subject": "M1.6: SiteEventLog — Deployment + Instance-lifecycle events", "status": "pending"},
{"id": 29, "subject": "M1.7: SiteEventLog — Store-and-Forward + Notification events", "status": "pending"},
{"id": 30, "subject": "M1.8: SiteEventLog — script started/completed (Info)", "status": "pending"},
{"id": 31, "subject": "M1.9: M1 integration verification + redeploy", "status": "pending", "blockedBy": [23, 24, 25, 26, 27, 28, 29, 30]},
{"id": 13, "subject": "M2 — Correctness & behavioral gaps (Tier 2) [umbrella; split per-item at execution]", "status": "pending"},
{"id": 14, "subject": "M3 — Script trust boundary (Tier 1 #1-#2) [umbrella]", "status": "pending"},
{"id": 15, "subject": "M4 — Doc reconciliation (Tier 4) [umbrella]", "status": "pending"}
{"id": 22, "subject": "M1.0: Confirm proto/site surface for audit pull (spike)", "status": "completed"},
{"id": 23, "subject": "M1.1: Production gRPC IPullAuditEventsClient", "status": "completed", "blockedBy": [22]},
{"id": 24, "subject": "M1.2: Wire reconciliation + purge actors as central singletons", "status": "completed"},
{"id": 25, "subject": "M1.3: SiteCallAudit periodic reconciliation pull", "status": "completed"},
{"id": 26, "subject": "M1.4: SiteCallAudit daily terminal-row purge scheduler", "status": "completed"},
{"id": 27, "subject": "M1.5: SiteEventLog — emit Alarm events", "status": "completed"},
{"id": 28, "subject": "M1.6: SiteEventLog — Deployment + Instance-lifecycle events", "status": "completed"},
{"id": 29, "subject": "M1.7: SiteEventLog — Store-and-Forward + Notification events", "status": "completed"},
{"id": 30, "subject": "M1.8: SiteEventLog — script started/completed (Info)", "status": "completed"},
{"id": 31, "subject": "M1.9: M1 integration verification + redeploy", "status": "completed", "blockedBy": [23, 24, 25, 26, 27, 28, 29, 30]},
{"id": 13, "subject": "M2 — Correctness & behavioral gaps (Tier 2) [umbrella; split per-item at execution]", "status": "completed"},
{"id": 14, "subject": "M3 — Script trust boundary (Tier 1 #1-#2) [umbrella]", "status": "completed"},
{"id": 15, "subject": "M4 — Doc reconciliation (Tier 4) [umbrella]", "status": "completed"}
],
"lastUpdated": "2026-06-15"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-16-ipsen-mes-movein.md.tasks.json
+10 -9
View File
@@ -1,14 +1,15 @@
{
"planPath": "docs/plans/2026-06-16-ipsen-mes-movein.md",
"tasks": [
{"id": 1, "subject": "Task 1: InboundDatabaseHelper (read-only DB for inbound scripts)", "classification": "standard", "status": "pending"},
{"id": 2, "subject": "Task 2: Expose Database to inbound scripts (context + executor)", "classification": "high-risk", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Build + regression-test ScadaBridge", "classification": "small", "status": "pending", "blockedBy": [2]},
{"id": 4, "subject": "Task 4: Commit Component A", "classification": "trivial", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03", "classification": "high-risk", "status": "pending", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: Apply Component B - update inbound IpsenMESMoveIn script", "classification": "high-risk", "status": "pending", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Apply Component C - IpsenMoveIn template script on T1", "classification": "high-risk", "status": "pending", "blockedBy": [5]},
{"id": 8, "subject": "Task 8: On-box end-to-end verification", "classification": "standard", "status": "pending", "blockedBy": [6, 7]}
{"id": 1, "subject": "Task 1: InboundDatabaseHelper (read-only DB for inbound scripts)", "classification": "standard", "status": "completed"},
{"id": 2, "subject": "Task 2: Expose Database to inbound scripts (context + executor)", "classification": "high-risk", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Build + regression-test ScadaBridge", "classification": "small", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "Task 4: Commit Component A", "classification": "trivial", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03", "classification": "high-risk", "status": "completed", "note": "done off-repo on wonder-app-vd03 (live methods authored against Database.* fixed there 2026-06-25)", "blockedBy": [4]},
{"id": 6, "subject": "Task 6: Apply Component B - update inbound IpsenMESMoveIn script", "classification": "high-risk", "status": "completed", "note": "done off-repo on wonder-app-vd03 (live methods authored against Database.* fixed there 2026-06-25)", "blockedBy": [5]},
{"id": 7, "subject": "Task 7: Apply Component C - IpsenMoveIn template script on T1", "classification": "high-risk", "status": "pending", "note": "NEEDS LIVE vd03 verification — tracked in the 2026-08-01 pending-work task list", "blockedBy": [5]},
{"id": 8, "subject": "Task 8: On-box end-to-end verification", "classification": "standard", "status": "pending", "note": "NEEDS LIVE vd03 verification — tracked in the 2026-08-01 pending-work task list", "blockedBy": [6, 7]}
],
"lastUpdated": "2026-06-16"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-16-m5-audit-hardening.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{
"planPath": "docs/plans/2026-06-16-m5-audit-hardening.md",
"tasks": [
{"id": 119, "subject": "M5.1 (T8): CLI audit tree + tree endpoint", "status": "pending"},
{"id": 120, "subject": "M5.2 (T6): Per-node stuck-count KPIs", "status": "pending"},
{"id": 121, "subject": "M5.3 (T7): Structured response-capture increments", "status": "pending"},
{"id": 122, "subject": "M5.4 (T4): ParentExecutionId tag-cascade", "status": "pending"},
{"id": 123, "subject": "M5.5 (T3): Per-channel retention overrides", "status": "pending"},
{"id": 124, "subject": "M5.6 (T5): SourceNode sentinel backfill + runbook", "status": "pending", "blockedBy": [119]},
{"id": 125, "subject": "M5.7: M5 integration verification + docs", "status": "pending", "blockedBy": [119, 120, 121, 122, 123, 124]}
{"id": 119, "subject": "M5.1 (T8): CLI audit tree + tree endpoint", "status": "completed"},
{"id": 120, "subject": "M5.2 (T6): Per-node stuck-count KPIs", "status": "completed"},
{"id": 121, "subject": "M5.3 (T7): Structured response-capture increments", "status": "completed"},
{"id": 122, "subject": "M5.4 (T4): ParentExecutionId tag-cascade", "status": "completed"},
{"id": 123, "subject": "M5.5 (T3): Per-channel retention overrides", "status": "completed"},
{"id": 124, "subject": "M5.6 (T5): SourceNode sentinel backfill + runbook", "status": "completed", "blockedBy": [119]},
{"id": 125, "subject": "M5.7: M5 integration verification + docs", "status": "completed", "blockedBy": [119, 120, 121, 122, 123, 124]}
],
"lastUpdated": "2026-06-16"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-16-script-analysis-consolidation.md.tasks.json
+9 -8
View File
@@ -1,13 +1,14 @@
{
"planPath": "docs/plans/2026-06-16-script-analysis-consolidation.md",
"tasks": [
{"id": 103, "subject": "M3.0: Spike — compile-surface stub feasibility + package refs", "status": "pending"},
{"id": 104, "subject": "M3.1: Build ScriptAnalysis project (policy+validator+compiler+surfaces+tests)", "status": "pending", "blockedBy": [103]},
{"id": 105, "subject": "M3.2: TemplateEngine deploy gate → shared analyzer", "status": "pending", "blockedBy": [104]},
{"id": 106, "subject": "M3.3: SiteRuntime → shared analyzer + parity test", "status": "pending", "blockedBy": [104]},
{"id": 107, "subject": "M3.4: InboundAPI → shared analyzer", "status": "pending", "blockedBy": [104]},
{"id": 108, "subject": "M3.5: CentralUI → shared analyzer (keep markers + Test-Run host)", "status": "pending", "blockedBy": [104]},
{"id": 109, "subject": "M3.6: Integration verification + docs + fixture cleanup", "status": "pending", "blockedBy": [105, 106, 107, 108]}
{"id": 103, "subject": "M3.0: Spike — compile-surface stub feasibility + package refs", "status": "completed"},
{"id": 104, "subject": "M3.1: Build ScriptAnalysis project (policy+validator+compiler+surfaces+tests)", "status": "completed", "blockedBy": [103]},
{"id": 105, "subject": "M3.2: TemplateEngine deploy gate → shared analyzer", "status": "completed", "blockedBy": [104]},
{"id": 106, "subject": "M3.3: SiteRuntime → shared analyzer + parity test", "status": "completed", "blockedBy": [104]},
{"id": 107, "subject": "M3.4: InboundAPI → shared analyzer", "status": "completed", "blockedBy": [104]},
{"id": 108, "subject": "M3.5: CentralUI → shared analyzer (keep markers + Test-Run host)", "status": "completed", "blockedBy": [104]},
{"id": 109, "subject": "M3.6: Integration verification + docs + fixture cleanup", "status": "completed", "blockedBy": [105, 106, 107, 108]}
],
"lastUpdated": "2026-06-16"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-17-debugview-tabs-trees.md.tasks.json
+9 -8
View File
@@ -1,13 +1,13 @@
{
"planPath": "docs/plans/2026-06-17-debugview-tabs-trees.md",
"tasks": [
{"id": 170, "subject": "DV-1: Native-binding linkage — additive AlarmStateChanged contract chain", "classification": "high-risk", "status": "pending"},
{"id": 171, "subject": "DV-2: Site snapshot — placeholder rows for idle native source bindings", "classification": "standard", "status": "pending", "blockedBy": [170]},
{"id": 172, "subject": "DV-3: DebugTreeNode model + attribute-tree builder", "classification": "standard", "status": "pending", "blockedBy": [170]},
{"id": 173, "subject": "DV-4: Alarm-tree builder — computed leaves + native binding grouping + roll-up", "classification": "standard", "status": "pending", "blockedBy": [172, 170]},
{"id": 174, "subject": "DV-5: DebugView page — tabs + two TreeViews + in-place updates", "classification": "standard", "status": "pending", "blockedBy": [173]},
{"id": 175, "subject": "DV-6: Documentation — CentralUI + SiteRuntime + streaming contract", "classification": "small", "status": "pending", "blockedBy": [173]},
{"id": 176, "subject": "DV-7: Integration — full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "pending", "blockedBy": [174, 175, 171]}
{"id": 170, "subject": "DV-1: Native-binding linkage — additive AlarmStateChanged contract chain", "classification": "high-risk", "status": "completed"},
{"id": 171, "subject": "DV-2: Site snapshot — placeholder rows for idle native source bindings", "classification": "standard", "status": "completed", "blockedBy": [170]},
{"id": 172, "subject": "DV-3: DebugTreeNode model + attribute-tree builder", "classification": "standard", "status": "completed", "blockedBy": [170]},
{"id": 173, "subject": "DV-4: Alarm-tree builder — computed leaves + native binding grouping + roll-up", "classification": "standard", "status": "completed", "blockedBy": [172, 170]},
{"id": 174, "subject": "DV-5: DebugView page — tabs + two TreeViews + in-place updates", "classification": "standard", "status": "completed", "blockedBy": [173]},
{"id": 175, "subject": "DV-6: Documentation — CentralUI + SiteRuntime + streaming contract", "classification": "small", "status": "completed", "blockedBy": [173]},
{"id": 176, "subject": "DV-7: Integration — full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "completed", "blockedBy": [174, 175, 171]}
],
"waves": [
{"wave": 1, "tasks": [170]},
@@ -16,5 +16,6 @@
{"wave": 4, "tasks": [174, 175]},
{"wave": 5, "tasks": [176]}
],
"lastUpdated": "2026-06-17"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-17-m6-kpi-history.md.tasks.json
+19 -18
View File
@@ -1,23 +1,24 @@
{
"planPath": "docs/plans/2026-06-17-m6-kpi-history.md",
"tasks": [
{"id": 1, "subject": "K1: Foundation contracts (Commons)", "nativeId": 136, "classification": "high-risk", "status": "pending"},
{"id": 2, "subject": "K2: Persistence — EF config + repository + migration", "nativeId": 137, "classification": "high-risk", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "K3: KpiHistory project scaffold + options", "nativeId": 138, "classification": "standard", "status": "pending", "blockedBy": [1]},
{"id": 4, "subject": "K4: KpiHistoryRecorderActor", "nativeId": 139, "classification": "high-risk", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "K5: Host wiring + appsettings", "nativeId": 140, "classification": "high-risk", "status": "pending", "blockedBy": [4]},
{"id": 6, "subject": "K6: NotificationOutboxKpiSampleSource", "nativeId": 141, "classification": "small", "status": "pending", "blockedBy": [1]},
{"id": 7, "subject": "K7: SiteCallAuditKpiSampleSource", "nativeId": 142, "classification": "small", "status": "pending", "blockedBy": [1]},
{"id": 8, "subject": "K8: AuditLogKpiSampleSource", "nativeId": 143, "classification": "small", "status": "pending", "blockedBy": [1]},
{"id": 9, "subject": "K9: SiteHealthKpiSampleSource", "nativeId": 144, "classification": "standard", "status": "pending", "blockedBy": [1]},
{"id": 10, "subject": "K10: KpiSeriesBucketer (pure helper)", "nativeId": 145, "classification": "small", "status": "pending", "blockedBy": [1]},
{"id": 11, "subject": "K11: KpiHistoryQueryService (CentralUI)", "nativeId": 146, "classification": "standard", "status": "pending", "blockedBy": [2, 10]},
{"id": 12, "subject": "K12: KpiTrendChart.razor reusable SVG component", "nativeId": 147, "classification": "standard", "status": "pending", "blockedBy": [1]},
{"id": 13, "subject": "K13: Notification Outbox page trend section", "nativeId": 148, "classification": "standard", "status": "pending", "blockedBy": [11, 12]},
{"id": 14, "subject": "K14: Site Calls page trend section", "nativeId": 149, "classification": "standard", "status": "pending", "blockedBy": [11, 12]},
{"id": 15, "subject": "K15: Audit Log page trend section", "nativeId": 150, "classification": "standard", "status": "pending", "blockedBy": [11, 12]},
{"id": 16, "subject": "K16: Health dashboard per-site trend panel", "nativeId": 151, "classification": "standard", "status": "pending", "blockedBy": [11, 12]},
{"id": 17, "subject": "K17: Integration — docs, deploy, Playwright, full verification", "nativeId": 152, "classification": "high-risk", "status": "pending", "blockedBy": [5, 6, 7, 8, 9, 13, 14, 15, 16]}
{"id": 1, "subject": "K1: Foundation contracts (Commons)", "nativeId": 136, "classification": "high-risk", "status": "completed"},
{"id": 2, "subject": "K2: Persistence — EF config + repository + migration", "nativeId": 137, "classification": "high-risk", "status": "completed", "blockedBy": [1]},
{"id": 3, "subject": "K3: KpiHistory project scaffold + options", "nativeId": 138, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 4, "subject": "K4: KpiHistoryRecorderActor", "nativeId": 139, "classification": "high-risk", "status": "completed", "blockedBy": [3]},
{"id": 5, "subject": "K5: Host wiring + appsettings", "nativeId": 140, "classification": "high-risk", "status": "completed", "blockedBy": [4]},
{"id": 6, "subject": "K6: NotificationOutboxKpiSampleSource", "nativeId": 141, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 7, "subject": "K7: SiteCallAuditKpiSampleSource", "nativeId": 142, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 8, "subject": "K8: AuditLogKpiSampleSource", "nativeId": 143, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 9, "subject": "K9: SiteHealthKpiSampleSource", "nativeId": 144, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 10, "subject": "K10: KpiSeriesBucketer (pure helper)", "nativeId": 145, "classification": "small", "status": "completed", "blockedBy": [1]},
{"id": 11, "subject": "K11: KpiHistoryQueryService (CentralUI)", "nativeId": 146, "classification": "standard", "status": "completed", "blockedBy": [2, 10]},
{"id": 12, "subject": "K12: KpiTrendChart.razor reusable SVG component", "nativeId": 147, "classification": "standard", "status": "completed", "blockedBy": [1]},
{"id": 13, "subject": "K13: Notification Outbox page trend section", "nativeId": 148, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 14, "subject": "K14: Site Calls page trend section", "nativeId": 149, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 15, "subject": "K15: Audit Log page trend section", "nativeId": 150, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 16, "subject": "K16: Health dashboard per-site trend panel", "nativeId": 151, "classification": "standard", "status": "completed", "blockedBy": [11, 12]},
{"id": 17, "subject": "K17: Integration — docs, deploy, Playwright, full verification", "nativeId": 152, "classification": "high-risk", "status": "completed", "blockedBy": [5, 6, 7, 8, 9, 13, 14, 15, 16]}
],
"lastUpdated": "2026-06-17"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md.tasks.json
+3 -2
View File
@@ -11,7 +11,8 @@
],
"reviewPolishCommit": "8dcc55f",
"followUps": [
{"id": 162, "subject": "normalize routed GetAttributes List values for cross-process transport (surfaced by WS-4 review)"}
{"id": 162, "subject": "normalize routed GetAttributes List values for cross-process transport (surfaced by WS-4 review)", "status": "resolved", "note": "RESOLVED — DeploymentManagerActor.cs RouteInboundApiGetAttributes now normalizes via NormalizeRoutedReturnValue"}
],
"lastUpdated": "2026-06-17"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md
+3 -1
View File
@@ -239,7 +239,9 @@ fully cover the DELMIA/MES use case.
> **Status:** IMPLEMENTED. `Route.To(code).WaitForAttribute(name, targetValue, timeout)` is wired
> end-to-end (`RouteToWaitForAttributeRequest/Response` → `IInstanceRouter` → `CommunicationService`
> → `SiteCommunicationActor` → `DeploymentManagerActor` → `InstanceActor`), value-equality only
> across the wire. NOT wired into the CentralUI Test-Run sandbox — that remains a follow-up.
> across the wire. ~~NOT wired into the CentralUI Test-Run sandbox — that remains a follow-up.~~
> **Follow-up SHIPPED 2026-07-10** — the CentralUI Test-Run sandbox is wired via
> `SandboxInstanceGateway`/`SandboxScriptHost` (deferred-work register row #14 resolved).
---
docs/plans/2026-06-17-waitfor-deferred-items.md.tasks.json
+6 -5
View File
@@ -1,10 +1,11 @@
{
"planPath": "docs/plans/2026-06-17-waitfor-deferred-items.md",
"tasks": [
{"id": 1, "subject": "WD-1: site-local WaitForAsync + WaitResult + quality-gated mode (§3+§4.2)", "classification": "high-risk", "status": "pending", "parallelizableWith": [2]},
{"id": 2, "subject": "WD-2a: routed contract + central path (§6 part 1)", "classification": "high-risk", "status": "pending", "parallelizableWith": [1]},
{"id": 3, "subject": "WD-2b: site unpacking + DeploymentManager handler (§6 part 2)", "classification": "high-risk", "status": "pending", "blockedBy": [2]},
{"id": 4, "subject": "WD-3: integration — docs + full verification", "classification": "standard", "status": "pending", "blockedBy": [1, 2, 3]}
{"id": 1, "subject": "WD-1: site-local WaitForAsync + WaitResult + quality-gated mode (§3+§4.2)", "classification": "high-risk", "status": "completed", "parallelizableWith": [2]},
{"id": 2, "subject": "WD-2a: routed contract + central path (§6 part 1)", "classification": "high-risk", "status": "completed", "parallelizableWith": [1]},
{"id": 3, "subject": "WD-2b: site unpacking + DeploymentManager handler (§6 part 2)", "classification": "high-risk", "status": "completed", "blockedBy": [2]},
{"id": 4, "subject": "WD-3: integration — docs + full verification", "classification": "standard", "status": "completed", "blockedBy": [1, 2, 3]}
],
"lastUpdated": "2026-06-17"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-18-m10-uiux-platform.md
+8 -8
View File
@@ -459,15 +459,15 @@ Markup: two `<input type="datetime-local">` with `id="@(IdPrefix)-from"`/`-to`,
## Deferred / out of scope (log as FOLLOWUPs at INT)
- Unified offset+keyset pagination framework (blocked by total-count mismatch).
- `Deployments.razor` PagerWindow → OffsetPager (different windowed UX; intentionally left).
- Complex `TemplateEdit` page-embedded modals → host migration.
- TreeView arrow-key navigation (R7).
- ~~Complex `TemplateEdit` page-embedded modals → host migration.~~ **DONE 2026-08-01.**
- ~~TreeView arrow-key navigation (R7).~~ **DONE 2026-08-01.**
- Theme-package side-rail dark theming, IF the spike verdict is "rail stays light" (coordination follow-up).
## Follow-ups logged at delivery (INT findings)
- **#207 (pre-existing, open since M6/K14):** `QueryStringDrillInTests` fixture does not register `IKpiHistoryQueryService` — 3 `SiteCallsReport` drill-in tests red on this gap; unrelated to M10.
- **#163 (pre-existing):** `InstanceConfigureListOverrideTests` codec roundtrip red; pre-dates M10.
- **NotificationReport `OffsetPager` always-visible:** pager is now always visible when results exist (previously hidden on sub-page-size sets); buttons are correctly disabled on a single page — product decision whether to re-add an `@if (_totalCount > _pageSize)` guard.
- **#207 — FIXED (2026-08-01 bookkeeping sync).** ~~(pre-existing, open since M6/K14):~~ `QueryStringDrillInTests` now registers the `IKpiHistoryQueryService` substitute in its fixture (lines ~165-167), so the 3 `SiteCallsReport` drill-in tests are green.
- **#163 — CLOSED (2026-08-01).** ~~(pre-existing):~~ `InstanceConfigureListOverrideTests` codec roundtrip verified green (6/6; `dotnet test …CentralUI.Tests --filter InstanceConfigureListOverride`) — fixed at some point since it was logged; no longer a known red. (#207 `QueryStringDrillInTests` re-verified green the same day, 4/4.)
- **NotificationReport `OffsetPager` always-visible — DECIDED + DONE (2026-08-01).** A pager whose only controls are permanently disabled is noise, so the conventional behaviour ships: the bar is hidden on a provably single page. Implemented **reusably but opt-in** — `OffsetPager.HideWhenSinglePage` (default `false`, so no consumer changes behaviour implicitly), set `true` at the `NotificationReport` call site. It is NOT defaulted on because the pager's summary span doubles as the "N total" result-count readout on `ConfigurationAuditLog`, which has no other place to show it (and whose Playwright fixture asserts `3 total` on a 3-row single page). The guard only fires when single-page-ness is *positively* established (`PageCount <= 1` **and** `Page <= 1` **and** `!HasNextPage`), so a null `TotalCount` still renders the bar.
- **`Deployments.razor` PagerWindow intentionally kept:** windowed numbered-button UX is deliberate; NOT migrated to `OffsetPager`.
- **`TemplateEdit` inline modals NOT migrated:** the page-embedded modals (wave-3 T34c already tokenized their backdrops); full migration to the host is deferred.
- **TreeView full arrow-key navigation (R7) still deferred.**
- **Full-app `bg-light`/`bg-white` → theme-aware utility sweep deferred:** only the bounded modal-surface offenders were addressed in T34c; INT dark-mode smoke may surface additional instances.
- **`TemplateEdit` inline modals — MIGRATED (2026-08-01).** All four page-embedded modals (Attribute, Alarm, Native Alarm Source, Script) now open through `IDialogService.ShowAsync`, so the host owns the backdrop, focus trap, Escape and focus restoration. Each body was extracted to its own component next to the page — `TemplateAttributeDialog`, `TemplateAlarmDialog`, `TemplateNativeAlarmSourceDialog`, `TemplateScriptDialog` — following the `MoveDataConnectionDialog` pattern: the body owns the form state and renders validation/server errors INLINE while staying open, closing with `Close(true)` only on a successful save. Persistence stayed on the page (it owns `TemplateService` and the inherited-member rules) and is reached through an `OnSaveAsync` delegate returning `null` on success or the message to display. The script dialog keeps all four tab panels mounted (Monaco/JSONJoy must not tear down on tab switch) and hosts the Test Run panel, which needs the live unsaved buffer. Extraction moved markup that three structural source-scanning tests pinned (`TemplateNativeAlarmSourceEditorTests`, `AttributeListEditorTests`, `TestRunWarningTests`) — all three were repointed at the new files.
- **~~TreeView full arrow-key navigation (R7) still deferred.~~ DONE 2026-08-01** — the WAI-ARIA tree pattern (roving tabindex, Arrow/Home/End movement, Enter/Space activation, `aria-level`) now ships on `TreeView<TItem>`; see `docs/components/TreeView.md`.
- **Full-app `bg-light`/`bg-white` → theme-aware utility sweep — DONE 2026-08-01.** 35 class swaps across 19 Central UI files: surfaces/`<pre>`/`<code>` `bg-light` → `bg-body-secondary`, panel `bg-white` → `bg-body`, neutral `badge bg-light text-dark` → `badge bg-secondary-subtle text-secondary-emphasis`, `bg-light text-muted border` → `bg-body-secondary text-body-secondary border`. **Deliberately left** (7 sites): the neutral member of a *status-badge* switch/ternary whose siblings are all solid non-theme-aware colours (`bg-success`/`bg-danger`/`bg-warning`), where swapping only the neutral one breaks the set's visual weight — `Topology.razor:513,588`, `InstanceConfigure.razor:1520`, `NotificationReport.razor` `StatusBadgeClass` fallback, `TransportImport.razor` ConflictKind fallback, `SecuredWrites.razor` `text-bg-light` fallback, `Health.razor:377`. `SchemaBuilder.razor:88` was already `bg-light-subtle` (theme-aware). `bg-dark text-light` console panels and `site.css`/`#reconnect-modal` are intentional fixed colours.
docs/plans/2026-06-18-m10-uiux-platform.md.tasks.json
+19 -18
View File
@@ -1,23 +1,24 @@
{
"planPath": "docs/plans/2026-06-18-m10-uiux-platform.md",
"tasks": [
{"id": 269, "subject": "M10-T33a: Modal host ShowAsync<T> + focus trap/restore + backdrop hook", "status": "pending"},
{"id": 270, "subject": "M10-T34-spike: Verify ZB.MOM.WW.Theme dark-mode feasibility", "status": "pending"},
{"id": 271, "subject": "M10-T41: Alarm-override Playwright trigger-config scenarios", "status": "pending"},
{"id": 272, "subject": "M10-T33b: Migrate 5 simple dialogs to ShowAsync<T>", "status": "pending", "blockedBy": [269]},
{"id": 273, "subject": "M10-T34a: Dark token layer in site.css + .sb-modal-backdrop", "status": "pending", "blockedBy": [270]},
{"id": 274, "subject": "M10-T35a: OffsetPager component", "status": "pending"},
{"id": 275, "subject": "M10-T35b: KeysetPager component", "status": "pending"},
{"id": 276, "subject": "M10-T35c: DateTimeRangeFilter component", "status": "pending"},
{"id": 277, "subject": "M10-T35d: Adopt OffsetPager+DateTimeRangeFilter into NotificationReport (+backdrop token)", "status": "pending", "blockedBy": [274, 276, 273]},
{"id": 278, "subject": "M10-T35e: Adopt OffsetPager+DateTimeRangeFilter into ConfigurationAuditLog", "status": "pending", "blockedBy": [274, 276]},
{"id": 279, "subject": "M10-T35f: Adopt KeysetPager+DateTimeRangeFilter into SiteCallsReport (+backdrop token)", "status": "pending", "blockedBy": [275, 276, 273]},
{"id": 280, "subject": "M10-T35g: Adopt KeysetPager into AuditResultsGrid + DateTimeRangeFilter into AuditFilterBar", "status": "pending", "blockedBy": [275, 276]},
{"id": 281, "subject": "M10-T35h: Adopt DateTimeRangeFilter into EventLogs", "status": "pending", "blockedBy": [276]},
{"id": 282, "subject": "M10-T34c: Tokenize remaining standalone backdrops + bg-* audit", "status": "pending", "blockedBy": [273]},
{"id": 283, "subject": "M10-T34b: Dark-mode toggle + persistence + SSR hydration", "status": "pending", "blockedBy": [270, 273]},
{"id": 284, "subject": "M10-T36a: Accessibility pass — TreeView chevron + icon audit + toast verify", "status": "pending"},
{"id": 285, "subject": "M10-INT: Integration — build, docker, Playwright, a11y+dark smoke, docs, review", "status": "pending", "blockedBy": [271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284]}
{"id": 269, "subject": "M10-T33a: Modal host ShowAsync<T> + focus trap/restore + backdrop hook", "status": "completed"},
{"id": 270, "subject": "M10-T34-spike: Verify ZB.MOM.WW.Theme dark-mode feasibility", "status": "completed"},
{"id": 271, "subject": "M10-T41: Alarm-override Playwright trigger-config scenarios", "status": "completed"},
{"id": 272, "subject": "M10-T33b: Migrate 5 simple dialogs to ShowAsync<T>", "status": "completed", "blockedBy": [269]},
{"id": 273, "subject": "M10-T34a: Dark token layer in site.css + .sb-modal-backdrop", "status": "completed", "blockedBy": [270]},
{"id": 274, "subject": "M10-T35a: OffsetPager component", "status": "completed"},
{"id": 275, "subject": "M10-T35b: KeysetPager component", "status": "completed"},
{"id": 276, "subject": "M10-T35c: DateTimeRangeFilter component", "status": "completed"},
{"id": 277, "subject": "M10-T35d: Adopt OffsetPager+DateTimeRangeFilter into NotificationReport (+backdrop token)", "status": "completed", "blockedBy": [274, 276, 273]},
{"id": 278, "subject": "M10-T35e: Adopt OffsetPager+DateTimeRangeFilter into ConfigurationAuditLog", "status": "completed", "blockedBy": [274, 276]},
{"id": 279, "subject": "M10-T35f: Adopt KeysetPager+DateTimeRangeFilter into SiteCallsReport (+backdrop token)", "status": "completed", "blockedBy": [275, 276, 273]},
{"id": 280, "subject": "M10-T35g: Adopt KeysetPager into AuditResultsGrid + DateTimeRangeFilter into AuditFilterBar", "status": "completed", "blockedBy": [275, 276]},
{"id": 281, "subject": "M10-T35h: Adopt DateTimeRangeFilter into EventLogs", "status": "completed", "blockedBy": [276]},
{"id": 282, "subject": "M10-T34c: Tokenize remaining standalone backdrops + bg-* audit", "status": "completed", "blockedBy": [273]},
{"id": 283, "subject": "M10-T34b: Dark-mode toggle + persistence + SSR hydration", "status": "completed", "blockedBy": [270, 273]},
{"id": 284, "subject": "M10-T36a: Accessibility pass — TreeView chevron + icon audit + toast verify", "status": "completed"},
{"id": 285, "subject": "M10-INT: Integration — build, docker, Playwright, a11y+dark smoke, docs, review", "status": "completed", "blockedBy": [271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284]}
],
"lastUpdated": "2026-06-18"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-18-m7-opcua-mxgateway-ux.md.tasks.json
+24 -23
View File
@@ -4,28 +4,28 @@
"branch": "worktree-m7-opcua-mxgateway-ux",
"baseSha": "254e0e7",
"tasks": [
{"id": 184, "subject": "M7-A1: Extract AlarmStateBadges shared component", "classification": "standard", "status": "pending"},
{"id": 185, "subject": "M7-A2: Operator Alarm Summary page + fan-out service", "classification": "standard", "status": "pending", "blockedBy": [184]},
{"id": 186, "subject": "M7-A3: Operator + Verifier roles + policies + LDAP mapping", "classification": "high-risk", "status": "pending"},
{"id": 187, "subject": "M7-B1: Browse type-info fields on BrowseNode", "classification": "standard", "status": "pending"},
{"id": 188, "subject": "M7-B2: BrowseNext continuation through browse contract", "classification": "high-risk", "status": "pending", "blockedBy": [187]},
{"id": 189, "subject": "M7-B3: Thread continuation token through browse plumbing", "classification": "standard", "status": "pending", "blockedBy": [188]},
{"id": 190, "subject": "M7-B4: Bounded recursive address-space search — adapter", "classification": "high-risk", "status": "pending", "blockedBy": [188]},
{"id": 191, "subject": "M7-B5: Search plumbing — message + actor + comm + service", "classification": "standard", "status": "pending", "blockedBy": [189, 190]},
{"id": 192, "subject": "M7-B6: NodeBrowserDialog — load-more + search box + type column", "classification": "standard", "status": "pending", "blockedBy": [189, 191]},
{"id": 193, "subject": "M7-B7: Verify-endpoint — message + site probe handler", "classification": "high-risk", "status": "pending", "blockedBy": [191]},
{"id": 194, "subject": "M7-B8: Verify-endpoint plumbing + UI", "classification": "standard", "status": "pending", "blockedBy": [193]},
{"id": 195, "subject": "M7-B9: Cert trust — per-node CertStore actor + broadcast", "classification": "high-risk", "status": "pending", "blockedBy": [193]},
{"id": 196, "subject": "M7-B10: Cert trust plumbing + cert-management UI", "classification": "standard", "status": "pending", "blockedBy": [194, 195]},
{"id": 197, "subject": "M7-C1: PendingSecuredWrite entity + persistence + migration", "classification": "high-risk", "status": "pending"},
{"id": 198, "subject": "M7-C2: Secured-write commands + submit/reject/list handlers", "classification": "high-risk", "status": "pending", "blockedBy": [197, 186]},
{"id": 199, "subject": "M7-C3: Approve → site write relay", "classification": "high-risk", "status": "pending", "blockedBy": [198]},
{"id": 200, "subject": "M7-C4: AuditKind.SecuredWrite + audit wiring", "classification": "high-risk", "status": "pending", "blockedBy": [198, 199]},
{"id": 201, "subject": "M7-C5: Secured Writes Central UI page", "classification": "standard", "status": "pending", "blockedBy": [198, 199, 186]},
{"id": 202, "subject": "M7-D1: OverrideCsvParser pure helper", "classification": "standard", "status": "pending"},
{"id": 203, "subject": "M7-D2: InstanceConfigure CSV import UI", "classification": "standard", "status": "pending", "blockedBy": [202]},
{"id": 204, "subject": "M7-D3: CLI instance import-overrides --file", "classification": "small", "status": "pending", "blockedBy": [202]},
{"id": 205, "subject": "M7-E1: Integration — docs, full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "pending", "blockedBy": [184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204]}
{"id": 184, "subject": "M7-A1: Extract AlarmStateBadges shared component", "classification": "standard", "status": "completed"},
{"id": 185, "subject": "M7-A2: Operator Alarm Summary page + fan-out service", "classification": "standard", "status": "completed", "blockedBy": [184]},
{"id": 186, "subject": "M7-A3: Operator + Verifier roles + policies + LDAP mapping", "classification": "high-risk", "status": "completed"},
{"id": 187, "subject": "M7-B1: Browse type-info fields on BrowseNode", "classification": "standard", "status": "completed"},
{"id": 188, "subject": "M7-B2: BrowseNext continuation through browse contract", "classification": "high-risk", "status": "completed", "blockedBy": [187]},
{"id": 189, "subject": "M7-B3: Thread continuation token through browse plumbing", "classification": "standard", "status": "completed", "blockedBy": [188]},
{"id": 190, "subject": "M7-B4: Bounded recursive address-space search — adapter", "classification": "high-risk", "status": "completed", "blockedBy": [188]},
{"id": 191, "subject": "M7-B5: Search plumbing — message + actor + comm + service", "classification": "standard", "status": "completed", "blockedBy": [189, 190]},
{"id": 192, "subject": "M7-B6: NodeBrowserDialog — load-more + search box + type column", "classification": "standard", "status": "completed", "blockedBy": [189, 191]},
{"id": 193, "subject": "M7-B7: Verify-endpoint — message + site probe handler", "classification": "high-risk", "status": "completed", "blockedBy": [191]},
{"id": 194, "subject": "M7-B8: Verify-endpoint plumbing + UI", "classification": "standard", "status": "completed", "blockedBy": [193]},
{"id": 195, "subject": "M7-B9: Cert trust — per-node CertStore actor + broadcast", "classification": "high-risk", "status": "completed", "blockedBy": [193]},
{"id": 196, "subject": "M7-B10: Cert trust plumbing + cert-management UI", "classification": "standard", "status": "completed", "blockedBy": [194, 195]},
{"id": 197, "subject": "M7-C1: PendingSecuredWrite entity + persistence + migration", "classification": "high-risk", "status": "completed"},
{"id": 198, "subject": "M7-C2: Secured-write commands + submit/reject/list handlers", "classification": "high-risk", "status": "completed", "blockedBy": [197, 186]},
{"id": 199, "subject": "M7-C3: Approve → site write relay", "classification": "high-risk", "status": "completed", "blockedBy": [198]},
{"id": 200, "subject": "M7-C4: AuditKind.SecuredWrite + audit wiring", "classification": "high-risk", "status": "completed", "blockedBy": [198, 199]},
{"id": 201, "subject": "M7-C5: Secured Writes Central UI page", "classification": "standard", "status": "completed", "blockedBy": [198, 199, 186]},
{"id": 202, "subject": "M7-D1: OverrideCsvParser pure helper", "classification": "standard", "status": "completed"},
{"id": 203, "subject": "M7-D2: InstanceConfigure CSV import UI", "classification": "standard", "status": "completed", "blockedBy": [202]},
{"id": 204, "subject": "M7-D3: CLI instance import-overrides --file", "classification": "small", "status": "completed", "blockedBy": [202]},
{"id": 205, "subject": "M7-E1: Integration — docs, full build, docker rebuild, Playwright, smoke", "classification": "high-risk", "status": "completed", "blockedBy": [184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204]}
],
"waves": {
"A": [184, 185, 186],
@@ -34,5 +34,6 @@
"D": [202, 203, 204],
"E": [205]
},
"lastUpdated": "2026-06-18"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}
docs/plans/2026-06-18-m8-transport.md.tasks.json
+17 -16
View File
@@ -1,21 +1,22 @@
{
"planPath": "docs/plans/2026-06-18-m8-transport.md",
"tasks": [
{"id": 210, "subject": "M8-A1: Commons name-map types + preview/selection/summary extensions", "status": "pending"},
{"id": 211, "subject": "M8-A2: Transport bundle DTOs — Site/DataConnection/Instance + BundleContentDto", "status": "pending"},
{"id": 212, "subject": "M8-A3: LineDiffer — pure Myers per-line diff helper (T20)", "status": "pending"},
{"id": 213, "subject": "M8-B1: DependencyResolver site/instance expansion", "status": "pending", "blockedBy": [211]},
{"id": 214, "subject": "M8-B2: EntitySerializer site/connection/instance mapping (both directions)", "status": "pending", "blockedBy": [211]},
{"id": 215, "subject": "M8-B3: ManifestBuilder + summary counts + schemaVersion 1.1", "status": "pending", "blockedBy": [210]},
{"id": 216, "subject": "M8-B4: Export plumbing — selection wiring, command, ManagementActor, CLI", "status": "pending", "blockedBy": [213, 214, 215]},
{"id": 217, "subject": "M8-C1: ArtifactDiff — Myers code diff (T20) + Site/Connection/Instance compares", "status": "pending", "blockedBy": [211, 212]},
{"id": 218, "subject": "M8-C2: BundleImporter.PreviewAsync — new-type diff + required-mapping detection + blockers", "status": "pending", "blockedBy": [214, 217]},
{"id": 219, "subject": "M8-D1: BundleImporter.ApplyAsync — nameMap, resolve-or-create, instance upsert + FK rewire", "status": "pending", "blockedBy": [218]},
{"id": 220, "subject": "M8-D2: #16 — real stale-instance enumeration in ImportResult", "status": "pending", "blockedBy": [219]},
{"id": 221, "subject": "M8-D3: Import plumbing — name-map through command, ManagementActor, CLI", "status": "pending", "blockedBy": [219, 216]},
{"id": 222, "subject": "M8-E1: Export wizard — Sites/Instances selection", "status": "pending", "blockedBy": [216]},
{"id": 223, "subject": "M8-E2: Import wizard — Map step + Modified +/- diff render", "status": "pending", "blockedBy": [221]},
{"id": 224, "subject": "M8-INT: Docs, full build, docker rebuild, Playwright, live smoke, end-to-end trace", "status": "pending", "blockedBy": [222, 223]}
{"id": 210, "subject": "M8-A1: Commons name-map types + preview/selection/summary extensions", "status": "completed"},
{"id": 211, "subject": "M8-A2: Transport bundle DTOs — Site/DataConnection/Instance + BundleContentDto", "status": "completed"},
{"id": 212, "subject": "M8-A3: LineDiffer — pure Myers per-line diff helper (T20)", "status": "completed"},
{"id": 213, "subject": "M8-B1: DependencyResolver site/instance expansion", "status": "completed", "blockedBy": [211]},
{"id": 214, "subject": "M8-B2: EntitySerializer site/connection/instance mapping (both directions)", "status": "completed", "blockedBy": [211]},
{"id": 215, "subject": "M8-B3: ManifestBuilder + summary counts + schemaVersion 1.1", "status": "completed", "blockedBy": [210]},
{"id": 216, "subject": "M8-B4: Export plumbing — selection wiring, command, ManagementActor, CLI", "status": "completed", "blockedBy": [213, 214, 215]},
{"id": 217, "subject": "M8-C1: ArtifactDiff — Myers code diff (T20) + Site/Connection/Instance compares", "status": "completed", "blockedBy": [211, 212]},
{"id": 218, "subject": "M8-C2: BundleImporter.PreviewAsync — new-type diff + required-mapping detection + blockers", "status": "completed", "blockedBy": [214, 217]},
{"id": 219, "subject": "M8-D1: BundleImporter.ApplyAsync — nameMap, resolve-or-create, instance upsert + FK rewire", "status": "completed", "blockedBy": [218]},
{"id": 220, "subject": "M8-D2: #16 — real stale-instance enumeration in ImportResult", "status": "completed", "blockedBy": [219]},
{"id": 221, "subject": "M8-D3: Import plumbing — name-map through command, ManagementActor, CLI", "status": "completed", "blockedBy": [219, 216]},
{"id": 222, "subject": "M8-E1: Export wizard — Sites/Instances selection", "status": "completed", "blockedBy": [216]},
{"id": 223, "subject": "M8-E2: Import wizard — Map step + Modified +/- diff render", "status": "completed", "blockedBy": [221]},
{"id": 224, "subject": "M8-INT: Docs, full build, docker rebuild, Playwright, live smoke, end-to-end trace", "status": "completed", "blockedBy": [222, 223]}
],
"lastUpdated": "2026-06-18"
"lastUpdated": "2026-08-01",
"status": "2026-08-01 bookkeeping sync: statuses reconciled against merged code"
}

Some files were not shown because too many files have changed in this diff Show More