Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Rig-only posture, set in the tracked docker/central-node-{a,b}/appsettings.Central.json
so it survives deploy.sh / container recreates (it used to be a throwaway compose
override on central-a alone). Both nodes so Traefik on :9000 is login-free
regardless of which node is active. POST /auth/login still answers 302, so the
Playwright fixture login is unaffected; LoginTests will fail on this rig by design.
Documented in the scadabridge-cluster-ops skill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Both central nodes gain a root AuditLog:PerChannelRetentionDays:ScriptTrace=30 (validator
floor). verify-delmia-notifier.sh step H: finds DelmiaRecipeDownload via api-method list,
runs flow-debug run-method with the drill parameters and asserts completed=true plus the
kind set {InboundRequest, RouteCall, ScriptRun, AttributeWrite, AttributeWait} from each
event's detailsJson, then flow-debug replay on the root ExecutionId with the same check,
retried up to 40 s for the 5 s site telemetry push; prints 'H live+replay -> YES'.
VERIFICATION PENDING: the image scadabridge:latest was built from this branch
(docker/build.sh, exit 0) but the containers have not been recreated and step H has not
run yet — the next step is a recreate through the MAIN checkout's compose file (deploy.sh
refuses a worktree, register row 68: bind-mounted appsettings would rehome), a fresh
DelmiaRecipeDownload-scoped key, and the drill.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
A caller addressing a standby central node directly (or hitting Traefik inside its
/health/active probe window after a failover) used to get 503 STANDBY_NODE. The
standby now relays the request verbatim to the active peer over HTTP
(IStandbyRequestForwarder / HttpStandbyRequestForwarder) and replays the response;
method scripts and Route.To() still execute only on the active node.
- ScadaBridge:InboundApi:StandbyForwardPeerEndpoints (static list of the OTHER
central node's HTTP base URL; empty = old 503 behaviour) + StandbyForwardTimeout
(60s), both startup-validated.
- Peers tried in order: peer 503 stamped X-ScadaBridge-Standby, connection failure
or timeout → next peer; all fail → 503 ACTIVE_NODE_UNREACHABLE. A 503 without
the marker is the active node's own answer and is replayed.
- One-hop loop guard via X-ScadaBridge-Forwarded; 413 body cap now runs BEFORE
the active-node check so a standby never buffers/relays an oversized body.
- AuditWriteMiddleware skips its own ApiInbound row for a relayed request so a
call is audited once, on the node that executed it.
- Rig + env2 central nodes configured; spec (Component-InboundAPI.md → Standby
forwarding), Component-Host.md options row, CLAUDE.md updated.
- 20 new tests (filter decision + relay mechanics + validator).
Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
The row read 'central nodes expose no Prometheus endpoint'. That is not quite
what was wrong, and the difference matters. Central already maps /metrics
(Program.cs central branch) and telemetry is registered for both roles through
the shared BindSharedOptions, so the endpoint rides endpoint routing and has
always been served on central's web port (:5000 on the rig, anonymous — there
is no FallbackPolicy). What central lacked was the site branch's dedicated
MetricsPort, which is the port the established live-scrape recipe probes:
docker exec <node> bash -c 'exec 3<>/dev/tcp/localhost/8084 ...'
So a probe of central on 8084 failed and central-only series (KPI/outbox
counters, the bundle-import session gauge + its new rejection counter,
alarm-cache gauges) read as unscrapeable while they were in fact being
exported on a different port.
Central now binds a dedicated HTTP/1.1 listener on ScadaBridge:Node:MetricsPort,
so one probe recipe covers all eight rig nodes. HTTP/1.1-capable because a
standard Prometheus scraper cannot negotiate h2c, which is why this cannot
share the gRPC port.
Read as a PRESENCE check, not GetValue's defaulted read. NodeOptions defaults
MetricsPort to 8084, so a defaulted read would make every existing central node
start binding a new port on upgrade — and a port already in use is a hard boot
failure, not a degraded one. Central gains the listener only where an operator
asked for it; site nodes are unchanged and still bind theirs unconditionally.
The bind is also skipped when the port would collide with the gRPC port or an
ASPNETCORE_URLS port, since Kestrel throws on a duplicate ListenAnyIP and
/metrics is reachable on those ports anyway.
Config only — no host port mapping added, because the probe runs inside the
container. Applied to both docker/ centrals and both docker-env2/ centrals.
FILE EDITS ONLY: no docker command was run, and deploy.sh was not invoked.
The rig pointed at the shared 10.100.0.35 GLAuth whose serviceaccount password
was rotated (SEC-36), so central login had been failing ('Authentication service
is misconfigured') and a TEMP DisableLogin workaround was pending. Central nodes
now point at the local redundant pair (scadaproj/infra/glauth-redundant,
host.docker.internal:3893 + FallbackServers :3894), where the dev bind password
is correct — live-gated on the redeployed rig: login OK, primary-kill failover,
sticky preference (bind-count proven), walk-back on backup-kill.
AuthFlowTests factory bound as cn=admin for search-then-bind, but the current
directory grants the search capability only to serviceaccount (admin searches
return 50 Insufficient access) — stale since the GLAuth config evolved; the test
had been skipping on the closed port and failed once anything answered :3893.
Now binds as serviceaccount; AuthFlowTests 5/5 against the pair.
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Populates ScadaBridge:Communication:CentralFetchBaseUrl in every central
appsettings for the notify-and-fetch deploy flow. An empty value now
causes a fail-fast on deploy; this config prevents that regression.
- docker/: http://scadabridge-traefik (in-network Traefik LB, port 80)
- docker-env2/: http://scadabridge-env2-traefik (env2 Traefik LB, port 80)
- src/Host base: http://localhost:5000 (ASP.NET Core default for single-host dev)
- deploy/wonder-app-vd03: http://localhost:8085 (gitignored; edited in main repo)
RUNBOOK Upgrading section updated with note on this setting.
Wire the M6 KPI History recorder into the central composition path:
- Program.cs: call services.AddKpiHistory(configuration) on the central-only
branch alongside AddNotificationOutbox/AddAuditLog/AddSiteCallAudit.
- AkkaHostedService.cs: register KpiHistoryRecorderActor as a central,
non-role-scoped ClusterSingletonManager + ClusterSingletonProxy + a
PhaseClusterLeave CoordinatedShutdown graceful-stop drain (singleton name
'kpi-history-recorder'), copied/adapted from the audit-log-purge block.
- appsettings.Central.json (Host + docker + docker-env2 central nodes): add a
ScadaBridge:KpiHistory section (SampleInterval 00:01:00, RetentionDays 90,
PurgeInterval 1.00:00:00, DefaultMaxSeriesPoints 200).
KPI history is observability/best-effort and MUST NOT gate readiness: the
recorder is deliberately NOT added to RequiredSingletonsHealthCheck or any
other readiness gate.
Adds a "Dev Disable-Login Flag" subsection to Component-Security.md covering
ScadaBridge:Security:Auth:DisableLogin / User, the AutoLoginAuthenticationHandler
mechanism, and the no-environment-guard / startup-warning production risk.
Ships DisableLogin: false under ScadaBridge → Security → Auth in:
- src/.../Host/appsettings.json (canonical default)
- docker/central-node-a/appsettings.Central.json
- docker/central-node-b/appsettings.Central.json
Also records DL-3 commit SHAs in the plan tasks file.
Both :9000 (docker) and :9100 (docker-env2) central nodes now bind the shared dev
GLAuth (scadaproj/infra/glauth/, dc=zb,dc=local) via the cn=serviceaccount search
account instead of the bundled scadabridge-ldap container (now commented out in
infra/docker-compose.yml, kept for rollback). Verified: multi-role -> all 4 roles
on both clusters with scadabridge-ldap stopped.
Replace dc=scadabridge,dc=local with dc=zb,dc=local in all dev/test LDAP
references — app config, docker test-cluster node configs (docker/ and
docker-env2/), GLAuth fixture, dev tooling, Host.Tests fixtures,
IntegrationTests factory, and operational test_infra docs. OU structure
(ou=SCADA-Admins,ou=users,etc.) preserved throughout. Email domains
(@scadabridge.local), hostnames, and container names are untouched.
Historical plan docs (2026-05-24-second-environment.md,
2026-05-31-folder-repo-rename-scadabridge-design.md) excluded as
point-in-time records. No synthetic dc=example,dc=com placeholders touched.
- NavMenu: move Import Bundle out of the nested RequireDesign/RequireAdmin
double-gate into the top-level Admin section so an Admin-only user sees it
without needing the Design role; Export Bundle stays in the Design section.
- TransportImport: inject IAuditService + ScadaLinkDbContext; emit a
BundleImportUnlockFailed audit row (best-effort, swallowed on failure) on
every wrong-passphrase attempt in SubmitPassphraseAsync, with attempt
number and error reason in afterState.
- docker central-node-a/b appsettings: add ScadaLink:Transport section with
SourceEnvironment = "docker-cluster" so the importer picks up a non-null
environment name in the audit trail.
- CentralUI.Tests: register IAuditService mock + SQLite in-memory
ScadaLinkDbContext in TransportImportPageTests to satisfy the two new injects.
Adds "NodeName" to the ScadaLink:Node section of each per-node
appsettings:
- central-a, central-b for the two central nodes
- node-a, node-b under each of the three sites (site-a, site-b, site-c)
After this commit + a redeploy, every fresh AuditLog / Notifications /
SiteCalls row gets stamped with the originating node's role name via
INodeIdentityProvider, satisfying the design's SourceNode invariant
end-to-end.
Both central nodes ran on the NotificationOutboxOptions code defaults
(100 / 10s = 600/min) because the mounted per-node appsettings.Central.json
had no ScadaLink:NotificationOutbox section. Add the section with
DispatchBatchSize 1000 + DispatchInterval 5s — measured ~6,000/min after
restart (sweep duration becomes the binding constraint, which is fine:
the no-overlap guard self-regulates). Dev-cluster tuning only.
The cookie SecurePolicy was hard-coded to Always, so the auth cookie was always
marked Secure and the browser never sent it over plain HTTP — making login
impossible on the HTTP-only Docker dev cluster (login succeeded server-side but
every following request was unauthenticated). Add SecurityOptions.RequireHttps-
Cookie (default true — production stays HTTPS-only); when false the cookie uses
SameAsRequest. The docker/ central nodes set it false.
Multi-stage Dockerfile with NuGet restore layer caching, per-node appsettings
with Docker hostnames, shared bridge network with infra services, and
build/deploy/teardown scripts. Ports use 90xx block to avoid conflicts.
Joseph Doherty