Commit Graph

3313 Commits

Author SHA1 Message Date
Joseph Doherty efade2eab0 feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)
SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo
LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and
keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair
(central + site, kill and stop-and-hold) before the flip.

- ClusterOptions default + validator/HOCON comments updated.
- Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/
  node appsettings now say leader-auto-down. The leader-auto-down overlays are
  kept as no-ops (drill records reference them); new docker-compose.auto-down.yml
  overlays keep Akka AutoDowning available for comparison drills.
- Tests: default asserts leader-auto-down; new HOCON test for an unset strategy;
  TwoNodeClusterFixture defaults to the production posture.
- Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster,
  README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed.

No deploy needed: DEV, QA and both rigs set the key explicitly.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:57:45 -05:00
Joseph Doherty 1213a74353 deploy(#116): DEV vd03 central + site on leader-auto-down (config only)
One key per file: ScadaBridge:Cluster:SplitBrainResolverStrategy auto-down ->
leader-auto-down in appsettings.Central.json and appsettings.Site.json, build
061432c8 unchanged. G6 boot lines on both services, health 200, MxGateway and
Z28062 at baseline, bogus key 401, Overview 6/6 Up. Downtime ~23 s central,
~30 s site. No drill: single-node clusters.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:46:47 -05:00
Joseph Doherty 50a5291b64 deploy(#143): DEV vd03 + QA vq03/vq04 to 061432c8 — hash chain removed (migration 65), #116 QA site-pair drills
DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central.
Chain schema + chainer role gone, env key removed, all gates pass on both tiers.
#116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold →
[leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:36:04 -05:00
Joseph Doherty 061432c8c4 deploy(#52/#142): Z28061 redeployed on QA (idle, verified); DEV skipped — idle attributes Bad quality
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 18:08:30 -05:00
Joseph Doherty 997e36a2bc deploy(#116): QA stop-and-hold drill proves the leader-auto-down timer path
vq04 (active central) killed with SCM auto-restart disabled and held down 94 s:
UNREACHABLE +11.4 s, [leader-auto-down] DOWNING +26.4 s, survivor vq03 active
+26.7 s. Recovery config restored byte-identical before Start-Service; vq04 rejoined
as standby in ~4.7 s with no exit-75. Also corrects the earlier Drill A 'sampler gap'
(an awk keying error; vq04 was ready at +20.8 s).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 18:02:21 -05:00
Joseph Doherty cae5ab958c deploy(#116): QA central kill drill (G14 central) — takeover ~10 s via new-incarnation path
Both centrals killed in turn (Stop-Process on the service PID). SCM restart in ~5 s;
the survivor downed the old incarnation at ~+9-10 s and took the active role; the
victim rejoined as standby at ~+21 s. leader-auto-down stood aside (already Down),
so its 15 s timer path was not exercised on QA. Site-pair drill still pending
(Z28061 slot, #142).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:53:09 -05:00
Joseph Doherty 3dea146c03 Merge branch 'chore/remove-audit-hash-chain': remove the AuditLog hash chain and two-login hardening (IT security, 2026-11-15)
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:51:07 -05:00
Joseph Doherty b418408c0f docs(audit): record the removal of the hash chain and the two-login hardening
Component-AuditLog (Security & Tamper-Evidence now a removal note, archive format v2, config),
Component-CLI, Component-ManagementService, HighLevelReqs AL-11/AL-12, docs/components/CLI, the CLI
README, alog.md §11.4/§15 and CLAUDE.md. New production script docs/plans/sql/RemoveAuditLogHashChain.sql
(idempotent, from migration 64); AddAuditLogHashChain.sql marked superseded. Historical plans,
deployment records and deploy-prep packs are left as written.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:38:22 -05:00
Joseph Doherty ed2ed0a367 feat(audit)!: remove the AuditLog hash chain, head export and two-login maintenance hardening
Owner decision 2026-11-15, at IT security's request, pending a family-wide general solution.

Removed: the chaining service, hasher, verifier, chain head/tombstone entities and store, the #139
chain-start cutoff, AuditLog:HashChain (incl. ChainBacklog and HeadExport with its File/Syslog sinks
and export service), the audit-chain-head-export health check, the VerifyAuditChain /
VerifyAuditChainHeads management commands and the `audit verify-chain` / `audit verify-heads` CLI
commands, and the two-login plumbing (AuditLog:MaintenanceConnectionString, AuditMaintenanceContext,
the keyed audit-maintenance repository/archive). Purge, archive, retention, partition roll-forward
and backfill-source-node all run on the runtime connection again, as before #107.

Parquet archive (#42) kept, chain-free: format v2 (ten stored columns, no tombstone file, manifest
without chain state), the switch guard is the row count only (51003 gone), verify = data-file size,
SHA-256 and row count, and superseded-stale cleanup proves coverage by EventId. Format-v1 archives
still list, verify and are cleaned up (their tombstone file too).

Migration RemoveAuditLogHashChain drops the chain indexes, the scadabridge_audit_chainer role
(members first), both chain tables, ChainSeq/RowHash and the purger's AuditMaintenanceGrants grants;
writer/purger roles stay. Down restores them. Every chain-column reference is EXEC-deferred for the
idempotent script (#126); FlipAuditLogClusteredKey's two chain index statements get the same
deferral, behaviour unchanged, so re-running the idempotent script on a migrated database works.
The partition-switch staging table adds the chain columns only when the live table has them, so the
new binary purges either schema. Deploy gate: old binaries map the chain columns, so the migration
ships with the new binaries, both centrals stopped.

A leftover AuditLog:HashChain section or MaintenanceConnectionString (DEV/QA still carry
AuditLog__HashChain__Enabled=true) is ignored with one startup warning (RetiredAuditSettingsWarning).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:38:22 -05:00
Joseph Doherty 125db03232 deploy(#52): DelmiaReceiver child-name fix applied on DEV + QA (Z28061 excluded)
CvdReactor.ProcessRecipeDownload now addresses Children["DelmiaReceiver"] on DEV vd03
and QA vq03/vq04. Redeployed DEV Z28062 + Z28061Sim, QA Z28062-Z28069; Z28061 left
stale on both tiers (shared with another team, no slot agreed). Traced read-only call
on Z28062 (ReadyFlag False): the pre-fix 'DelmiaRecv.ReadyFlag not found' warning is
gone after the fix.

apply.sh: every CLI call passes --url/--username/--password explicitly (an env-only
SCADABRIDGE_URL silently reached DEV), plus SB_EXPECT (refuse unless the expected
instances exist = tier proof) and SB_SKIP (never deploy the listed instances).

Record: docs/deployment/deployment-records/2026-11-15-gate-52-delmiareceiver-fix.md

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 17:25:37 -05:00
Joseph Doherty 6489c9fc0f Guard dialog completion by presentation identity 2026-11-13 21:25:42 -05:00
Joseph Doherty d370dbd37c fix(ui): preserve precise audit submissions and adopt delivered secrets 2026-11-13 19:42:53 -05:00
Joseph Doherty 9dbfe7bcaf refactor(ui): prioritize permitted operational views and touch topology 2026-11-13 19:24:12 -05:00
Joseph Doherty 0c41b613ac refactor(ui): unify semantic presentation and table hierarchy 2026-11-13 19:06:18 -05:00
Joseph Doherty 2d30323ae8 Validate audit windows and isolate dashboard count reads 2026-11-13 18:23:12 -05:00
Joseph Doherty 102cbb2e90 Correct raw JSON accessibility regression fixture premise 2026-11-13 18:00:12 -05:00
Joseph Doherty 915a8a5914 Associate ScadaBridge controls, guidance and editor errors 2026-11-13 17:59:00 -05:00
Joseph Doherty 20885b6161 Adopt published Theme 1.0.1 reflow patch 2026-11-13 17:42:14 -05:00
Joseph Doherty 87c882ec0a Name dialogs and contain native overlay keyboard focus 2026-11-13 17:42:14 -05:00
Joseph Doherty cdd3adf6fc Adopt Theme 1.0.0 with supported dark palette 2026-11-13 16:12:40 -05:00
Joseph Doherty f8c2851d25 test(ui): verify bUnit 2.7.2 with patched parser 2026-11-13 15:23:32 -05:00
Joseph Doherty c4ec51e89a fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary
reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the
next unreadable sample cleared the streak, and the connection retried the stopped
node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s.

Two going-away triggers on top of #138, both still requiring a Healthy (>= 200),
higher level on the other endpoint and honouring the minimum dwell:
- ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead
  of waiting out the window - once per degraded streak, so a non-Healthy peer is
  re-probed once per window, not every poll. Other Degraded levels (standby 90/50,
  booting 1) still wait the window.
- A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable
  samples don't overwrite it) probes the other endpoint before the first
  reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect
  at zero delay, the stopped node is never retried; otherwise the ordinary retry
  cycle runs (less the probe's time). After only Healthy levels (250/240, Unset),
  inside the dwell, or with the feature off: unchanged retry-then-failover.

Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and
non-trigger with positive controls, connect-order artifact assertions) and a real
in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s
(lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound).
13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings.
Docs: Component-DataConnectionLayer, components/DataConnectionLayer.

Not deployed; QA live re-check pending.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:36:59 -05:00
Joseph Doherty b19836788b docs(deploy): record QA vq03/vq04 audit hash chain enablement (#107, #139)
SourceNode backfill 843 rows (0 NULL), AuditLog__HashChain__Enabled=true on both
centrals (ChainBacklog unset, head export off, single login), pair stopped together and
started vq03 first: central API ~29 s, sites not restarted. One anchor at
2026-11-13T14:07:43.3208169Z, 0 pre-cutoff rows chained, 20 new rows ChainSeq 1-20,
verify-chain 2026-11 Intact on both centrals, 2026-10 NotChained. MxGateway 2196 Good,
WarmCanary 2 Good, LocalDb sync connected. Backup ScadaBridge_pre-hashchain_20261113T090524.bak.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:13:00 -05:00
Joseph Doherty 933da568e3 docs(deploy): record QA WarmCanary OPC UA connection (vq03/vq04 OtOpcUa) and the #138 live gate
Connection 2 / template 27 / instance 16 on site-a, bound to the OtOpcUa script-runner
canaries; serviceLevelFailover at its defaults. Under Warm the connection moved off the
Standby by ServiceLevel (connect-time preference, 0.1 s) and on the booting Standby (50);
a graceful Primary stop still took the connectivity path (66 s, #141). Kept as monitor.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 09:01:19 -05:00
Joseph Doherty 89d2becba9 refactor(host): remove legacy site-psk-invalidation topic (#105)
Removes the #96 compatibility path that the unified central-cache-invalidation
topic (#100 item 3, 153609ef; compat 3a31dd2d) carried for one release:
- CentralCacheInvalidationBus no longer publishes a bare SitePskInvalidated
  on site-psk-invalidation; LegacySitePskTopic constant deleted.
- CentralCacheInvalidationSubscriber no longer subscribes to that topic or
  handles a bare SitePskInvalidated; it applies envelopes only.
central-cache-invalidation behaviour is unchanged (one envelope per
invalidation, origin filtering, per-handler fault isolation).

Gate met: 8e32f353 (carries 153609ef and 3a31dd2d) runs on every central
node - DEV vd03 (record 581e25ca) and QA vq03/vq04 (record 8ba5d5d7).

Mixed-version / rolling-upgrade constraint: a central node built before
153609ef listens only on site-psk-invalidation and no longer receives PSK
invalidations from a newer peer (only its TTL and the site-rejection refresh
bound the staleness). Mixed pairs across that boundary are no longer
supported; central pairs are stopped and started together anyway.
Documented in Component-Host.md and Component-Communication.md.

Tests: the legacy publish/subscribe tests are replaced by
PskRotation_OnOneNode_ReachesThePeersCaches_ThroughTheUnifiedTopic (node A
publishes, node B's PSK cache and secret-resolver cache are invalidated, and
nothing else is published) and a bare-message-ignored test with an envelope
positive control. The two-node cluster test now asserts the PSK invalidation
arrives exactly once and the publisher applies nothing.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 08:43:55 -05:00
Joseph Doherty 8ba5d5d7fd docs(deploy): record QA vq03/vq04 0e540291 -> 8e32f353, migrations 58->64, #116 leader-auto-down (#107, #116)
Pair window per pack 8ffa6394: COPY_ONLY backup + VERIFYONLY, stop both pairs,
idempotent EF script 58->64 from the Mac (flip in ~2 s), swap, centrals then
sites started together. Site data path ~2 min 6 s, central API ~58 s.
leader-auto-down started on all 4 nodes; no #116 drills; hash chain off;
recovery flag TRUE on all 4 (actions kept). Env digest-identical; only the
strategy key changed in appsettings; per-node JWT/idle values preserved.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 08:14:03 -05:00
Joseph Doherty 581e25caaf docs(deploy): record DEV vd03 424db155 -> 8e32f353, #131 applied, audit hash chain enabled (#107, #139)
No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132).
#131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after.
Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off),
one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:21:31 -05:00
Joseph Doherty 8ffa639477 docs(deploy): refresh QA vq03/vq04 pack to main 8e32f353 - no new migration, owner's go scope (#107, #116)
- Target 3a59c3f6 -> 8e32f353 (adds #138 ServiceLevel failover, #139 chain
  cutoff, #140 graceful stop, LocalDb 0.3.8). No migration in the range;
  the idempotent 58->64 script regenerated at 8e32f353 is byte-identical
  (SHA-256 488da93d...c34ecc0).
- Owner's go 2026-11-13 ("just deploy"): #116 leader-auto-down flip and the
  exit-75 service recovery setting in; no #116 drills; hash chain and head
  export off; #52 stays excluded.
- #131 re-confirmed DEV-only against the QA config DB.
- Preflight re-run 07:05 EST: 58 migrations, 3,441 AuditLog rows, EventId
  key, log 63 MB used of 200.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:11:46 -05:00
Joseph Doherty 8e32f3538c chore(deps): pin ZB.MOM.WW.LocalDb 0.3.8 (scadaproj#57)
LocalDb 0.3.8 ends open sync streams on ApplicationStopping and refuses
new ones Unavailable while stopping (lmxopcua#906, library side). The
host-side ShutdownStreamCancellationInterceptor (#140) stays.

GracefulStopWithOpenSyncStreamTests' positive control passed the stop
in ~900 ms under 0.3.8, because the library alone now ends the stream.
The control registers LocalDbSyncService without the host lifetime (the
0.3.7 behaviour), so it still reproduces the 8 s drain; verified it
fails on 0.3.8 without that change.

Release build 0 warnings. Host.Tests 733/733, IntegrationTests 133/133
(incl. the LocalDb site-pair convergence tests), Communication.Tests
843/843, HealthMonitoring.Tests 131/131, SiteEventLogging.Tests 118/118.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:08:59 -05:00
Joseph Doherty 1a6108c402 fix(host): leave the cluster before Kestrel drains and end open streams on stop (#140)
A graceful stop of a site node whose peer holds an open LocalDb sync stream
left the Akka cluster ~30 s late (the ScadaBridge half of lmxopcua#906,
scadaproj#57). WebApplication stops the web server before AkkaHostedService,
whose StopAsync ran CoordinatedShutdown, and Kestrel's graceful drain waited
the whole ShutdownTimeout for /localdb_sync.v1.LocalDbSync/Sync: LocalDb 0.3.7
observes only the call token, which Kestrel cancels at the END of the drain.
CancelAllStreams only covered SiteStreamGrpcServer's own streams.

- AkkaHostedService is now an IHostedLifecycleService. StoppingAsync runs
  before any hosted service's StopAsync (Kestrel's included): marks the stop
  intentional, raises ApplicationStopping (so CancelAllStreams still precedes
  actor teardown), then runs and awaits CoordinatedShutdown(ClrExitReason),
  bounded by the host shutdown token. StopAsync returns the same completed
  shutdown. Exit codes unchanged: planned stop exits 0; only a termination
  outside host shutdown (self-down / #132 shut-out) exits 75. No Akka.Hosting
  here, so no ClusterLeavingReason/-1 exit-code path is involved.
- ShutdownStreamCancellationInterceptor, innermost on both the central and the
  site AddGrpc pipelines: every streaming call's token is linked with
  ApplicationStopping; new streams are refused Unavailable while stopping.
  Covers the LocalDb sync stream until LocalDb 0.3.8 is pinned.

Evidence (IntegrationTests/GracefulStopWithOpenSyncStreamTests: real
WebApplication + Kestrel h2c + MapZbLocalDbSync behind LocalDbSyncAuthInterceptor
+ a real LocalDb SyncBackgroundService peer + the production AkkaHostedService
forming a single-member cluster, 8 s drain budget, IServer wrapped to timestamp
the drain): before 8859 ms stop, member removed at 8834 ms, drain began at 0 ms;
after 960 ms stop, member removed at 950 ms, drain began at 959 ms.

Mutations (all killed): StoppingAsync skips CoordinatedShutdown; interceptor
passes the original context (stop 8003 ms); no Unavailable refusal; StoppingAsync
does not raise ApplicationStopping; lifecycle interface dropped; site pipeline
loses the interceptor.

Release build of the solution: 0 warnings. Host.Tests 733/733,
IntegrationTests 133/133, Communication.Tests 843/843.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 04:16:42 -05:00
Joseph Doherty 3badd1f9fe docs(deploy): QA pack hash-chain enablement uses the #139 cutoff (52da02a7, no migration)
Section 7: the enablement needs a build containing 52da02a7 (3a59c3f6 would
still chain the backlog); keep ChainBacklog unset; verify the anchor and that
no pre-cutoff row is chained; enablement-only rollback.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:15:49 -05:00
Joseph Doherty 52da02a76a feat(audit): hash-chain cutoff - chain only rows written from first enablement, no backlog (#139)
Owner decision on #107 (2026-11-13): enabling AuditLog:HashChain must not
chain the historical backlog. main chained every unchained row oldest-first.

- The first chaining pass on a pristine database (no head, no chained row)
  writes a chain-start anchor: AuditLogChainTombstone (cutoff month,
  ChainSeq 0), PurgedAtUtc = DB SYSUTCDATETIME(), RowHash =
  SHA-256("ScadaBridge.AuditLog.ChainCutoff.v1" || YYYY-MM || ticks).
  Written once under the exclusive chain lock, so both centrals agree
  whatever their clocks/config. No migration: existing table + grants.
- Only rows with OccurredAtUtc >= cutoff are chained. The anchor hash is
  the cutoff month's genesis, binding the cutoff into the chain.
- AuditLog:HashChain:ChainBacklog (default false) restores chain-everything,
  read only on the very first enablement. Legacy chains (heads, no anchor)
  keep chaining everything.
- verify-chain splits unchained rows into UnchainedRows vs PreCutoffRows
  (+ ChainFromUtc); new breaks ChainedBeforeCutoff (row re-dated before the
  cutoff, or cutoff moved forward) and CutoffAnchorInvalid (anchor hash /
  month / multiplicity). Archive verify uses the anchor from its own
  tombstone file or the DB.
- Partition switch-out keeps the anchor (deletes ChainSeq > 0 only); the
  head store / export / verify-heads tombstone count excludes it.
- CLI verify-chain table prints Chain from + Pre-cutoff rows.

Tests: AuditLogHashChainCutoffTests (7, MSSQL), 6 verifier unit tests,
service/CLI/management wire tests; 9 mutations all killed.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:14:14 -05:00
Joseph Doherty 712c5117ee feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)
Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:06:54 -05:00
Joseph Doherty 16209831ca docs(deploy): QA vq03/vq04 upgrade pack 0e540291 -> 3a59c3f6, migrations 58 -> 64, #116 bundled (not executed) (#107)
Coordinated central-pair window for #107 item 4: range analysis, ordered
idempotent migration script (SHA-256 488da93d...), read-only SQL preflight
results from wonder-sql-vq03, procedure, gates, rollback, downtime estimate.
QA is on 0e540291 / 58 migrations (not b3021e7c / 57). Hash-chain enablement
after the window is gated on #139.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:50:26 -05:00
Joseph Doherty 3a59c3f638 test(overrides): case-variant override names verified end to end through the flattener; UI single path gets the hint (ScadaBridge#105)
Re-verified #105 item 2 on the real write paths: ManagementActor bulk attribute
overrides (CLI set-overrides / management API) and InstanceService.SetAttributeOverrideAsync
(Central UI single editor) both match the template member ordinally and refuse a
case-variant name, so nothing is stored for the flattener to drop. Native-source
overrides already validate against the flattener's own output.

- New theory tests drive each path, feed whatever it stored into the real
  FlatteningService, and assert every stored override is applied (exact-case row is
  the positive control). Mutation: accepting case variants (OrdinalIgnoreCase) fails
  them with "stored ... but the flattener dropped it as unresolved".
- The did-you-mean hint moves to TemplateEngine.OverrideNameMatching and is now also
  on the UI single-attribute path; ambiguous matches get no hint.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:13:10 -05:00
Joseph Doherty bba3c348bc docs(audit): AuditLogEntries index-usage measurement, window too short, no drop yet (#70)
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:11:03 -05:00
Joseph Doherty e2f433e849 feat(cli): read secret-bearing option values from a file or stdin (#137)
Every CLI option that can carry a secret gets a `--<name>-file <path|->`
sibling ('-' reads stdin), so the value stays out of the process list and
shell history: --password-file (global), --primary-config-file /
--backup-config-file (data-connection create|update), --auth-config-file
(external-system create|update), --connection-string-file (db-connection
create|update), --credentials-file (notification smtp create|update) and
--auth-token-file (notification sms create|update). Follows the existing
--config-file convention; `@path` is avoided because System.CommandLine
expands @-tokens as response files.

The value is read at parse time; a missing/unreadable/empty source, or '-'
on a terminal stdin, is a parse error naming option + path only, never the
content. Inline and file forms are mutually exclusive, at most one option
may read stdin, and one trailing line break is stripped. Inline forms keep
working.

docker/seed-sites.sh and docker-env2/seed-sites.sh now write the MxGateway
config (windev gateway key) to a mode-600 temp file removed by an EXIT trap
and pass --primary-config-file; seed-flow-outbound-demo.sh pipes its rig
connection string via --connection-string-file -. CLI README (new "Secrets
from a file or stdin" section + per-command tables), Component-CLI.md and
both docker READMEs updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:46:17 -05:00
Joseph Doherty 1e98997ce1 fix(communication): site pair log lines name the target site, not the logging node (#136)
The channel-pair log templates used {SiteId}, which collides with the
node's own enriched SiteId property, so the failover warning read
"Site central node A unavailable...". Renamed to {TargetSiteId} in all
six templates in SitePairChannelProvider.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:28:31 -05:00
Joseph Doherty fd80aeab57 fix(communication): fail a site command over within the call when the stopped node never received it (#136)
The first mutation (instance deploy) to a site pair whose sticky node was down
returned OUTCOME_UNKNOWN: SitePairChannelProvider.ExecuteAsync treated every
Unavailable on a non-retry-safe command as possibly-applied unless the server
sent the not-dispatched trailer, flipped stickiness, and threw. The retry then
succeeded on the live peer.

A connect-phase failure provably never sent the request, so it is now treated
like the pre-dispatch rejection: GrpcRetrySafety.WasNeverSent recognises
Grpc.Net.Client's subchannel connect failure ("Error connecting to subchannel."
with a SocketException cause) and an HttpRequestException name-resolution /
connection error with a connect-only socket error. Anything that may have been
delivered (reset, premature end, GOAWAY, server Unavailable) stays
OUTCOME_UNKNOWN and is not replayed. The site->central reconcile path uses the
same ProvablyNotApplied rule.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:22:54 -05:00
Joseph Doherty d7fa31f0b8 fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)
When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.

The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.

Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.

Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:11:56 -05:00
Joseph Doherty c3c1f6e870 chore(docker): windev MxGateway API key no longer in the seed scripts
docker/seed-sites.sh and docker-env2/seed-sites.sh hard-coded the windev
gateway API key. They now read MXGW_APIKEY from the environment, or from
a private file (default ~/.config/scadabridge/windev-mxgw-apikey,
override MXGW_APIKEY_FILE). With no key the MxGateway connection step is
skipped and the rest of the seed runs. READMEs note where to put it.

The key remains in git history (2 commits), so it should be rotated on
the windev gateway.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 00:00:03 -05:00
Joseph Doherty 644f564c30 docs(drill): env2 seeded data-path soak - site-x and central failovers with live data and S&F, no buffered loss (#116)
Unreplicated, the site-x survivor came up empty (by design: no LocalDb peer);
replicated, kill and freeze each lost only the 26-30 s takeover window, every
buffered message was delivered once, and the 4 duplicate ticks came from the
frozen victim's 6 s zombie window. Central kill: 0 gap.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:56:18 -05:00
Joseph Doherty 6c7e150a32 chore(docker-env2): replicate site-x's LocalDb, mirroring docker/ site-a (#116)
Unreplicated, the standby holds no deployed configs, data connections or
S&F buffer: a seeded active kill showed the survivor take over the cluster
in 28 s but load no instances and run no scripts. Dev-only sync key.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:42:26 -05:00
Joseph Doherty 8b6d326824 chore(docker-env2): seed-data-path.sh - opc-plc + read-only Galaxy instances and a store-and-forward sink for the #116 soak
Adds an env2-owned HTTP sink container (scadabridge-env2-sfsink) whose 503
mode makes ExternalSystem.CachedCall buffer, and an EnvSoak template whose 2 s
SoakTick sends one tick-numbered cached call, so loss and duplicates are
countable at the sink.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:42:26 -05:00
Joseph Doherty 90d20030a3 docs(drill): env2 record - tooling commit SHA after rebase (#116)
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:28:43 -05:00
Joseph Doherty cafdc0ce60 docs(drill): leader-auto-down env2 rollout - all #116 gates pass on central and site-x; env2 stays on it as the soak (#116)
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:28:34 -05:00
Joseph Doherty e8a743de9f chore(docker): env2 runs its own image tag, leader-auto-down overlay, drill targets env2 (#116)
- docker/build.sh: SCADABRIDGE_IMAGE picks the tag (default scadabridge:latest).
- docker-env2/docker-compose.yml: node image is ${SCADABRIDGE_IMAGE:-scadabridge:latest},
  so env2 can run its own build without retagging the primary rig's image.
- docker-env2/docker-compose.leader-auto-down.yml: opt-in overlay for both env2 pairs,
  mirroring docker/docker-compose.leader-auto-down.yml. Shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_ENV=primary|env2 (container prefix, Traefik :9100,
  pair site-x).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:28:34 -05:00
Joseph Doherty a92a418d7f docs(deploy): prepare the #52 / #131 site-script fixes for DEV and QA (not applied)
Read-only export of DEV vd03 and QA vq03 (2026-11-12): the only
Children["DelmiaRecv"] user is CvdReactor.ProcessRecipeDownload (script 4,
identical on both envs; DEV 3 / QA 9 instances), and the only ScriptAlarm
.Severity reader is DEV TestMesMachine.AlarmStatus (script 42, SBTEST01/02).
QA has no #131 work.

Adds the original (rollback) and fixed bodies, a guarded check/apply/rollback
script that re-sends every script field, and the apply plan: #52 any time,
#131 with or after the dd4f1c2a binary. The old #131 body still compiles on
main but returns banded values; the fixed body is what fails to compile on the
deployed binaries. Rig evidence for both gates is in the README.

Refs #52 #131

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:19:07 -05:00
Joseph Doherty a2e9d71824 feat(dcl): move the OPC UA client to OPC Foundation SDK 2.0.1 (#125)
OPCFoundation.NetStandard.Opc.Ua.Client and the test-only .Server move
1.5.378.182 -> 2.0.1, the version OtOpcUa master runs (lmxopcua#782; #799
and #801 closed). Transitive Core/Core.Types/Types/Configuration/
Security.Certificates resolve to 2.0.1.

API breaks fixed in the DCL adapter:
- Built-in types are immutable structs (Variant, DataValue, NodeId,
  LocalizedText, StatusCode, ArrayOf<T>). New OpcUaValueBoundary restores
  the SDK 1.5 CLR shapes for every read, subscription and alarm-event
  value (byte[], UTC DateTime with no-timestamp = MinValue, Guid,
  detached arrays) and builds write variants with SDK 1.5's
  new Variant(object) semantics (byte[] = ByteString scalar).
- StatusCodes members are no longer constants: switches/patterns over
  them (verify-outcome mapping, connection-level classification,
  BadResourceUnavailable) match by value.
- CertificateIdentifier no longer caches a certificate and the
  CertificateValidator/CertificateValidation event is gone: the
  configuration's CertificateManager owns trust lists and the loaded
  application certificate and is disposed with the session; auto-accept
  and the verify probe's capture-and-reject use AcceptError.
- new UserIdentity(X509Certificate2) is gone: X509 user identities use
  X509IdentityTokenHandler with an on-demand PFX provider (still fails
  fast on a missing file/key).
- DefaultSessionFactory takes the telemetry context; collections are
  ArrayOf<T>; EventFilter select clauses and ContentFilter operands are
  built immutably.

UserName identity (lmxopcua#880): SDK 2's default handler sends a
password over 64 bytes - every 65-character OtOpcUa API key - as an
RsaEncryptedSecret that SDK 1.5 servers reject with BadIdentityTokenInvalid
and SDK 2 servers accept only within a 5-minute clock window (#873). The
DCL now always uses LegacyRsaUserNameIdentityTokenHandler (ported from
OtOpcUa be28aa44), i.e. exactly what it sent on SDK 1.5, so it logs in to
both server generations during a mixed rollout.

Tests: the in-process secured test server is ported to SDK 2 and can
admit an API key; new tests cover the handler (mutation-checked: 5 of 7
fail with the SDK default handler), the value boundary, and a 65-char key
end to end over None/Sign/SignAndEncrypt with a wrong key reaching the
key check (BadIdentityTokenRejected).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:20:53 -05:00
Joseph Doherty 95584e2fcb chore(deps): ZB.MOM.WW.Telemetry(.Serilog) 0.3.0 -> 0.3.1 (scadaproj#56)
0.3.1 stops the gRPC/HttpClient instrumentation tracing telemetry's own
OTLP export calls (the Serilog log sink's LogsService/Export spans).
The app's own outgoing gRPC/HTTP spans are unchanged.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:16:33 -05:00