Plan only, nothing executed. Records owner decisions (mxgw on all four boxes,
current main via QA, NAM AD LDAPS day one, signed QA bundle), the SMB-verified
box facts (admin account, subnets, .NET, MXAccess 6000.0045.2657.1; PAR boxes
wait on the WinPlatform deploy), phases 0-8, port matrix, rollback and open
questions.
Blockers: no-role-no-login family standard (OtOpcUa Phase A refuses empty and mapper-throws logins,
reversing Login_when_role_mapper_throws_signs_in_with_no_role_claims; new ScadaBridge Phase A2);
OtOpcUa validates only the FallbackServers + BindMode/UserBindTemplate rules (the library validator
would block boot on DEV's empty ServiceAccountDn). Majors: inject the library logger; shared-account
lockout model, no-lockout ask, value verification, canary, stop rule; DEV automation identity; rolling
standby-first restarts for config-only; HG case-sensitive GroupToRole (optional Phase A3); mxgw
config-bound cert pin. Minors and nits applied; review disposition table in section 11.
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
- New plan docs/plans/2026-11-15-nam-ldap-cutover.md (supersedes the 2026-08-13 plan, now a pointer):
current state per app/tier, Phase A OtOpcUa fix (ToLibraryOptions drops FallbackServers/BindMode/
UserBindTemplate/AcceptDomainQualifiedNames; reflection completeness test), decisions D1-D7
(dedicated service account scadaproj#58 = QA prerequisite), winrm deploy-scope runbook, idempotent
ScadaBridge LdapGroupMappings SQL, corrections table.
- ldap_mapping.md rewritten as the canonical matrix under the rule 'one AD group grants exactly one
role per app'; ScadaBridge Deployer moves to a new UG_WW_APP_DEPLOY_<T> group (was a two-row insert
the unique index rejects).
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
LocalDbSyncService observed only the call token, which ASP.NET Core cancels
at the END of Kestrel's graceful drain. A peer's sync stream never ends on
its own, so every graceful stop of the passive node waited out the host's
whole ShutdownTimeout (30 s on OtOpcUa QA).
The service now takes an optional IHostApplicationLifetime (DI supplies it
in any generic host): ApplicationStopping is linked into the stream's token
and ends the session cleanly, and a stream arriving once stopping has begun
is refused Unavailable. Wire-compatible; no proto/schema change.
Tests: open stream ends within 5 s of StopApplication, new stream refused
Unavailable, control without a lifetime still running; both halves
mutation-killed. 262/262. Source only, NOT published (still 0.3.7).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Re-ran the two design 9 checks that were blocked for OtOpcUa on 2026-07-24, against a
docker-dev rig from OtOpcUa master 412da27c (Health 0.3.0) with the dashboard in a
container on the rig network:
1. one Active per pair: exactly one /health/active 200 per mesh (MAIN central-2,
SITE-A site-a-1, SITE-B site-b-1), dashboard shows one ACTIVE + one STANDBY each;
2. leader chip on every group; network-partitioning site-b-2 from site-b-1 produced
two 1-member clusters, both Active, and the SPLIT BRAIN chip, which cleared once
site-b-2 was restarted (healing the link alone does not merge two Akka clusters,
the documented OtOpcUa recovery); stopping site-a-1 moved the SITE-A chip.
Evidence (screenshots, /health/active bodies, timeline) under
docs/plans/evidence/2026-11-12-overview-phase4/. Ledger task 20 -> completed (20/20).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The Serilog OTLP log sink is not part of the OTel SDK and does not run
inside SuppressInstrumentationScope, so every log batch produced a root
opentelemetry.proto.collector.logs.v1.LogsService/Export gRPC client span.
AddZbTelemetry now drops telemetry's own transport from tracing:
- HttpClient instrumentation: FilterHttpRequestMessage rejects export calls
- gRPC client instrumentation (no Filter option): the start-time
EnrichWithHttpRequestMessage callback clears Recorded/IsAllDataRequested,
which also leaves the child HTTP activity unsampled.
An export call is any /opentelemetry.proto.collector.* path, or an
OTLP/HTTP /v1/{traces,metrics,logs} path on the configured endpoint's
host:port (ZbOtlpExportCallFilter). Every other outgoing call keeps its span.
Tests: predicate theory + an end-to-end run against a loopback OTLP
receiver with positive controls (app gRPC + HttpClient calls traced) and
negatives (the real log-sink export, /v1/logs, gRPC path over HttpClient);
mutation-checked three ways.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Metrics, traces and logs from both rig centrals with node.role=Central and site.id=central;
deployments.applied and inbound_api.requests arrive. Follow-ups scadaproj#56 (log-export
self-spans) and ScadaBridge#134 (rig image lacks git SHA).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Evidence doc gains the #55 section (root cause of no logs, 0.3.0 contents + SHA256,
per-app commits, collector output per app: service.version, site.id, node.role and
log records on OtOpcUa, HistorianGateway, ScadaBridge Site and MxAccessGateway).
GAPS, current-state, SPEC, the Telemetry CLAUDE.md and the index row updated;
infra/otel-collector gains otelidentity.py (Resource per signal + log samples).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Exporter, OtlpEndpoint, ServiceVersion, SiteId, NodeRole, ExportLogs and
ResourceAttributes:<key> from <App>:Telemetry, absent keys keep code defaults, so
the four apps stop re-implementing the exporter parse and gain the identity keys.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- service.version: null ServiceVersion now defaults to the entry assembly's
AssemblyInformationalVersion (the family apps stamp the git SHA there); "" omits it.
- ResourceAttributes: extra Resource attributes on every signal; first-class keys
(service.*, host.name, site.id, node.role) are reserved and rejected at startup.
- Logs over OTLP: AddZbTelemetry publishes its options in DI and AddZbSerilog's OTLP
log sink follows them when its own options do not select Otlp, so
<App>:Telemetry:Exporter=Otlp now switches logs along with metrics and traces, with
the same endpoint and Resource. ExportLogs=false opts out.
- Ships the unpublished 0.2.1 review fixes, including the stricter OTLP endpoint check
(scheme-less host:port rejected when Exporter=Otlp); no app config sets an endpoint.
- Tests 42 -> 74, incl. a loopback OTLP/gRPC receiver proving logs arrive with the
Resource, and an ExportLogs=false case with a metrics positive control.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ILdapGroupLookup.GetUserGroupsAsync(username, ct) re-reads a known user's
groups with the service account: same extraction, canonicalisation, endpoint
walk (FallbackServers), timeouts and TLS policy as AuthenticateAsync, via one
shared WalkEndpoints<T>. Explicit statuses: Found, UserNotFound, UserDisabled
(AD userAccountControl), AmbiguousUser, DirectoryUnavailable, NotSupported
(BindMode=User / no service account / foreign ILdapAuthService),
ProviderDisabled. AddZbLdapAuth registers it as the same singleton.
0.5.0 is the first published version carrying 0.4.2-0.4.4; scadaproj#51 was
live-verified against the NAM AD (24/24, identity check mutation-proven).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
QA had no historian account and now reuses DEV's personal NAM account
(follow-up historiangw#134). Env migration: RuntimeDb removed,
LiveWritesEnabled on, gRPC 0.0.0.0:5221 kept.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Both packages published to the feed 2026-11-12, packed from 01c6dc1 (the 0.2.0
code + corrected RepositoryUrl, before the 0.2.1 commit); SHA256 of the feed
download matches the local pack. ScadaBridge switched to AddZbSerilog with the
MinimumLevelGovernor + ConfigureLogger hooks (main 2d2e9f35) and deleted its
LoggerConfigurationFactory. Updates the Observability row, Telemetry/CLAUDE.md
and components/observability/GAPS.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Published ZB.MOM.WW.LocalDb, .Contracts and .Replication 0.3.7 to the
Gitea feed (carrying the 0.3.5/0.3.6 changes); SHA256 recorded. Pins:
ScadaBridge main 12fa7aff (ScadaBridge#130), OtOpcUa master e3031654
(lmxopcua#889). Not deployed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Self-signed cert (SAN IP 10.100.0.35), public cert checked in, key host-only.
Deployed on 10.100.0.35: TLS chain verifies with the cert and fails without
it; raw LDAPS simple bind returns 0 for a good password and 49 for a bad one;
plain :3893 bind unchanged. README: thumbprint, client config, rotation; also
fixes the user-bind DN example (GLAuth puts users under their primary group OU).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
.app-shell min-height is now 100vh with a 100dvh override (the 3.3rem was a
donor-app top-bar leftover; ThemeShell has no top bar) and the lg+ sticky
rail is 100vh/100dvh. Source bumped to 0.5.3, UNPUBLISHED. Adds a static
regression test and fixes the CS8602 in NavStateScriptTests (0 warnings).
Browser-checked via Overview on local 0.5.2 vs 0.5.3 packages: at 390x844 a
short page went from a 791px shell (53px gap) to 844px, no extra scroll;
1440x900 rail stays sticky full-height.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner chose option (b). ZB.MOM.WW.Secrets.Replicator.Grpc has had no consumer
since ScadaBridge retired the gRPC secrets hub (2026-09-28); OtOpcUa never used it.
- removed the project and its test project from ZB.MOM.WW.Secrets.slnx, so the
family build/test and build/push.sh's `dotnet pack` skip it;
- csproj IsPackable=false (a direct pack of the project yields nothing);
- SecretsGrpcHubServerExtensions / SecretsGrpcHubClientExtensions [Obsolete];
- README, CHANGELOG (0.6.6 Deprecated), CLAUDE.md and clustered-secrets.md say so.
Source + tests stay in the tree and still build/run standalone (test csproj
suppresses CS0618): 200/200. Feed versions 0.4.1-0.6.3 are left for the owner
to unlist.
Also zeroes the solution's pre-existing warnings: CA1416 on two Unix-only
permission tests (guarded by Skip.IfNot, which the analyzer cannot see — now
[UnsupportedOSPlatform("windows")]) and two xUnit2031 Assert.Single(Where).
Build 0 warnings; tests 429 passed / 21 skipped / 0 failed; pack = 5 nupkgs.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when
it is past TombstoneRetention AND the peer is known to hold it:
- held while database-wide or per-table snapshot debt is outstanding (the
snapshot is built from the row-version ledger, so a pruned tombstone is a
delete it can no longer carry — the issue's second path);
- a local tombstone held while its delete is still in the unacked oplog.
Delivery (delta ack covering the delete, or snapshot debt discharged) restamps
tombstone_utc so retention runs from delivery. New optional
LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention)
bounds an undelivered tombstone's life. No proto, lib_schema_version or
on-disk schema change.
Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC):
offline update on B, newer delete on A, A's clock +8 days (oplog age cap +
tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now;
plus a no-prune positive control and a pruned-after-delivery control, and five
controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Upstreams ScadaBridge's local .rail-scroll pane. ScrollNav (default false)
wraps Nav in .rail-scroll and adds side-rail-scroll; layout.css (lg+ only)
fills the rail and scrolls just the pane so .rail-foot stays pinned.
Default markup/CSS unchanged for other consumers. Stays at 0.5.2, unpublished.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-004–009, 011, 012, 014–016, 018 and 019 marked Resolved with commit, fix and regression
evidence; Secrets open findings 13 -> 0; code-reviews/README.md regenerated (--check passes).
CHANGELOG 0.6.6 records the fixes, behaviour changes and the #49 follow-ups.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI host takes its working directory as the content root, and the rig put the store under it.
The old lexical store-path rule missed that on macOS only because the temp path is spelled
/var/... while the working directory reports /private/var/...; with Secrets-005's link resolution
the rule (correctly) refused it. The rig now runs from a sibling work/ directory.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The reload gap for a plain IConfigurationRoot is documented rather than fixed: closing it would mean
inserting a provider into ConfigurationRoot's internal list. No family host is exposed (plain roots
are built with reloadOnChange: false; the rest pass a ConfigurationManager). clustered-secrets.md
notes that AddZbSecretsAkkaReplication now starts the actor itself, so OtOpcUa's
SecretReplicationStarter becomes redundant once it adopts 0.6.6.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-019: the hub channels are created with ThrowOperationCanceledOnCancellation = true, so a
call cancelled through its token throws OperationCanceledException. Defensively, the failover
reader rethrows RpcException(Cancelled)-with-a-cancelled-token as cancellation instead of dialing
every fallback with the same cancelled token, and the sweep treats it as shutdown, not a failure
recorded in ISecretsHubSyncStatus. A server-side Cancelled status (token not cancelled) still fails
over.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-018: BundleService.ImportAsync now rehydrates every entry and re-wraps every foreign-KEK
row before the first write, so a malformed or tampered entry anywhere fails the import with the
store untouched ("Nothing was imported"). A store fault during the apply phase (no multi-row
transaction) now reports the row it stopped at and every row already written, which the
interactive flow prints, instead of a bare "Import failed".
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-016: RevealButton (RevealPolicy), SecretEditor and ConfirmDeleteModal (ManagePolicy) now
evaluate their policy through IAuthorizationService at the top of the revealing/mutating handler
and fail closed (no auth state, no IAuthorizationService, or a failed policy), instead of relying
on the host to wrap them in AuthorizeView. New tests build the REAL AddSecretsAuthorization
policies and evaluate each role against each policy, so secrets-manager satisfying RevealPolicy
would now fail a test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-015: RevealButton now writes the secret.reveal Success audit before showing the value and
shows nothing ("audit record could not be written") if that write throws; the failure is logged as
a warning with actor and name, never the value. The cancellation token is captured up front, so a
component disposed mid-reveal can no longer turn the audit into a swallowed ObjectDisposedException.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-014: SecretEditor in add mode reads the name first. A live row is refused ("already
exists. Use Rotate") with no write and no audit, so a pasted name can no longer replace a live
credential and reset its description/content type under a "secret.add" audit. A tombstoned name is
restored only when the operator submits the same name a second time after the warning.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-011: HandleAnnounceTick pipes the manifest read back to the actor and publishes the
announce from the mailbox with Self as sender. Under a genuinely async store it used to publish
from a pool continuation with NoSender, so every peer replied to deadLetters; a faulted manifest
read was dropped silently and is now logged.
Secrets-012: the exchange's push rows, the pull reply and the pull request are split into
messages of at most 64 KB (measured with the wire encoding), half Akka.Remote's default 128000-byte
frame, so a cold node converges against a large store. A single over-budget row is still sent
alone; the manifest announce is not split (documented in clustered-secrets.md).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-009: SecretReferenceExpander.ExpandAsync throws FormatException when a value holds a
${secret: marker that is not a complete reference with a non-blank name (${secret:}, an
unterminated ${secret:name, a nested marker), instead of passing the literal text through as the
credential. ExpandConfigurationAsync names the offending key (never the value) and writes nothing.
The README wiring sample drops the relative "SqlitePath": "secrets.db" (which failed validation)
for the per-user default and shows an absolute example; a test validates the sample.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI's own appsettings.json set "SqlitePath": "secrets.db", which the store-path validator
rejects as relative, so the headless CLI failed at startup when run from its own directory unless
Secrets__SqlitePath overrode it. The key is removed: the per-user default
(<LocalApplicationData>/ZB.MOM.WW/secrets.db) applies, and an app's appsettings.json in the working
directory or the environment still override it. Regression test validates the shipped file.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-008: SqlServerSecretSyncService.SweepAsync now catches a per-row push failure, logs it
(EventId 4, name only) and continues, like the pull side. One row the hub refuses (a >450-char
name, an over-long explicit KEK id) no longer stops every later name from reaching the hub.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-006: DefaultSecretResolver keeps a per-key invalidation generation; Invalidate bumps it
before evicting, and a resolve that read the store under an older generation withdraws its insert,
so a rotation mid-resolve is not undone for another TTL. Expired entries are removed on lookup and
by a periodic TimeProvider sweep (every TTL, at least 1 s); a non-positive TTL caches nothing.
The resolver is now IDisposable (stops the sweep timer).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-005: AddIfNotRooted now uses Path.IsPathFullyQualified, so Windows drive-relative
(C:secrets.db) and root-relative (\secrets.db) paths are rejected. AddIfUnderContentRoot refuses a
store if EITHER its spelled location (device prefix stripped, loopback admin share mapped to its
drive) OR its resolved location (symlinks/junctions followed, 8.3 names expanded on Windows) is
under the likewise-resolved content root. A store spelled through the content root stays refused
even if a link inside carries it elsewhere. UNC paths through other shares are compared as written.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-004: AesGcmEnvelopeCipher.Decrypt now wraps the ArgumentException AesGcm throws for a
wrong-length nonce/tag/wrapped DEK as SecretDecryptionException (as Rewrap already did), so the
resolver's decryption-failed audit fires.
Secrets-007: new MasterKeySnapshot (key + KekId from one reading, zeroed on Dispose) and a
default-implemented IMasterKeyProvider.GetSnapshot(); MasterKeyProviderBase overrides it to
resolve the source once and derive the id from that same key. Encrypt/Decrypt/Rewrap take one
snapshot per KEK and dispose it, so a key file swapped mid-operation can no longer stamp a row
with one key's id and wrap it under another. Intermediate key buffers are zeroed. On-disk format
unchanged: a golden row written by the 0.6.5 cipher still decrypts.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- Over-length key records (key_id > 64, prefix > 16, display name > 256) are
refused with ArgumentException before connecting; an over-64 key-id lookup
or mutation answers not-found without querying (SqlClient silently
truncated the sized parameter, so a different id sharing the first 64
characters matched).
- Every key_id predicate adds an exact VARBINARY comparison, so lookups are
case- and trailing-space-exact like SQLite; the PK seek is unchanged and
there is no schema change.
- SqliteApiKeyStoreImporter reports case-only key-id collisions and
over-length rows in new init lists (KeyIdCollisions, KeysRejected) instead
of counting them as skips or importing them truncated.
- Audit text truncation kept deliberately and documented.
Offline refused-port tests and live SQL Server 2022 tests fail pre-fix;
SqlServer suite 101/101 live. Auth findings now 0 open; CLAUDE.md records
the #46/#50 work; code-reviews/README.md regenerated.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A key-store lookup that threw (shared SQL Server unreachable, SQLite I/O
fault) escaped VerifyAsync despite the "only exception path is cancellation"
contract, surfacing as an unaudited 500 / gRPC Unknown. ApiKeyFailure gains
StoreUnavailable (appended; numeric values unchanged) and the lookup maps any
non-cancellation exception to it. Unit theory and an offline refused-port
SqlServerApiKeyStore test fail pre-fix.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Auth-007's sAMAccountName match assumed the search base covers one AD
domain; across a forest (Global Catalog) it could accept another domain's
same-named account and return its groups. For a u:DOMAIN\name Who-am-I
answer the entry must now also be that domain's account: AD's
msDS-PrincipalName (objectSid-derived DOMAIN\name, read by a base search
through the new ILdapConnection.Read) must equal the answer where offered;
otherwise the name must have exactly one holder under SearchBase and it must
be the entry found. Three forest cases fail pre-fix; single-domain and
sAMAccountName-search compat cases pass both ways. README notes this and
Auth-009.
(scadaproj#50)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ExtractUserName returned the first value of a possibly multi-valued
UserNameAttribute (default cn), so jsmith could sign in as "John Smith"
depending on server ordering. It now prefers the value equal to the typed
name ignoring case (still canonicalising case) and falls back to the first
value only when none matches.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
README: 0.3.6 version heading and changelog entries (0.3.5, 0.3.6), OtOpcUa
Phases 1+2, Synchronous EXTRA, per-table baseline wording in Ops notes, test
count 249. CLAUDE.md records LocalDb-005/006/007 and the #50 items, with the
cross-repo LocalDbSyncAuthInterceptor decision left as a note. LocalDb
findings now 0 open; code-reviews/README.md regenerated.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- SnapshotApplier now takes LocalDbMetrics (wired by SyncSessionFactory) and
adds every row it dead-letters - poison, drift reject, FK orphan, retry
failure, cap spill - to localdb.sync.dead_lettered. Before, only the delta
path recorded the metric.
- New ReplicationOptions.MaxSnapshotDeferredRows (default 100,000, validated
> 0) bounds the rows held for a missing FK parent; past it the oldest get
one more apply with deferral off and are dead-lettered if still orphaned.
An older name-ordered sender could otherwise defer a whole child table.
Unit + 2-node (SyncSessionFactory) tests fail pre-fix. No wire change.
(scadaproj#50)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Each node assigns its own AUTOINCREMENT keys, so two different rows inserted
on the two nodes share a pk_json and LWW silently keeps one on both nodes.
RegisterReplicated now refuses such a table (detected from SQLite column
metadata, before any trigger is installed). A plain INTEGER PRIMARY KEY is
still accepted (application-assigned ids are legitimate); the requirement
that replicated keys be unique across both nodes is documented in the
RegisterReplicated remarks and README Limitations. No ScadaBridge or OtOpcUa
replicated table is AUTOINCREMENT.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A column name containing ' was captured fine but ended the JSON-path SQL
literal early on the apply and snapshot paths, dead-lettering every row of the
table on the peer and faulting the snapshot page query for a quoted PK column.
Both paths now share JsonPathSql.For, which doubles ' after the JSON-path
escape. 2-node delta and wiped-peer snapshot tests fail pre-fix.
(scadaproj#46)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The drift guard lived only in the delta path; SnapshotApplier applied every
row unchecked, so a future-stamped snapshot row was applied, observed by the
clock and persisted as last_seen_hlc - moving the receiver's HLC ahead across
restarts despite FailClosedOnDrift (LocalDb-004). SnapshotApplier now takes
ReplicationOptions and runs the same per-row check (warn once per snapshot;
dead-letter under FailClosedOnDrift via the shared HlcDriftGuard, with the
row's HLC kept out of last_seen_hlc).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
INSERT OR REPLACE resolved a secondary UNIQUE conflict by deleting a
different row (another PK) with capture suppressed and its ledger left live,
so a pair diverged silently (LocalDb-003). The applier now uses
INSERT ... ON CONFLICT(<pk>) DO UPDATE (DO NOTHING when every column is PK):
a clash on another unique key aborts the entry, which is dead-lettered, and
each node keeps its own row. Side effect: a remote update of a parent with
ON DELETE RESTRICT children now applies instead of being dead-lettered.
Also waits for the snapshot watermark in WipedPeer_SnapshotLiftsItsWatermark,
which raced SnapshotComplete under full-suite load.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The snapshot streamed tables by name and rows by pk_json text, so a child
(or a self-reference whose parent's key sorts later) reached the receiver
first, failed its immediate FK check and was dead-lettered - permanently,
since the snapshot's watermark and ack covered it (LocalDb-002).
Sender: stream referenced tables first (Kahn order, name tie-break).
Receiver: SnapshotApplier defers rows that fail only
SQLITE_CONSTRAINT_FOREIGNKEY, retries them as rows land, and dead-letters
true orphans at SnapshotComplete. Delta path unchanged; wire unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LocalDbSyncService never read request headers, so setting
LocalDb:Replication:ApiKey on both nodes left the passive endpoint open
(LocalDb-001). It now requires Authorization: Bearer <key> (constant-time)
before the handshake and before newest-wins takeover, refusing anything else
with Unauthenticated. A blank key keeps the old accept-all behaviour and
logs a startup warning. Wire-compatible: the initiator has always sent the
header when the key is set; ScadaBridge/OtOpcUa interceptors use the same key.
LocalDb 0.3.5 -> 0.3.6 (unpublished).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
BindMode=User bound as UserBindTemplate({0}) and then returned the groups of
whatever entry (UserNameAttribute={0}) found - two directory lookups that can
name two accounts (Auth-007). Reproduced live against GLAuth: mallory bound as
herself and signed in with boss's Admins group.
After the bind the service now asks RFC 4532 Who-am-I and requires the entry
to be that account (dn: -> DN match; AD u:DOMAIN\name -> sAMAccountName match).
Without a Who-am-I answer only a DN template is provable (bind DN must equal
the entry DN); anything else fails closed as GroupLookupFailed. {0} is
RFC 4514-escaped in DN templates; validator message leads with EXAMPLE\{0}.
Auth 0.4.3 -> 0.4.4 (unpublished).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
`secret set`/`rotate` took the plaintext as a positional argument (shell
history, process list), and Program handed argv to
Host.CreateApplicationBuilder, whose command-line config provider crashed on
a value like "-p=Hunter2Secret" with the value in the unhandled
FormatException on stderr.
- SetCommandLine: `set|rotate <name> [-|--value-stdin]`; no value reads piped
stdin or prompts twice (masked) at a TTY; one trailing newline dropped;
empty refused. The positional <value> still works and warns on stderr.
- host built from HostApplicationBuilderSettings without argv; startup
failures print `secret: startup failed: <msg>` and exit 1
- unexpected trailing arguments reported by position, not echoed
- tests: SetCommandLineTests incl. two child-process runs of the built CLI;
the -p=Hunter2Secret probe fails (leaks) on the old Program.cs
- docs: README, clustered-secrets runbook, CHANGELOG
- code-reviews: Secrets-017 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SecretEditor and ConfirmDeleteModal wrote the success audit inside the same
try as the store write, so a throwing IAuditWriter after a committed
add/rotate/delete showed an error, wrote a Failure audit and skipped the
resolver-cache eviction - the old plaintext kept resolving for the TTL.
The mutation try now covers only the cipher + store call; the success audit
is best-effort afterwards and logs a warning (no value) on failure, so the
change is reported saved, the callback fires and the cache is evicted.
- tests: AuditFailureAfterCommitTests (add, rotate, delete) with a throwing
success audit; all fail without the fix
- code-reviews: Secrets-013 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replication actor was created only when something resolved ISecretStore
(store -> replicator -> SecretReplicationActorProvider.ActorRef). A node that
expanded its ${secret:} references pre-host and never touched the store at
runtime had no actor: no topic subscription, no announces, no pull replies.
AddZbSecretsAkkaReplication now registers SecretReplicationActorStarter, which
creates the actor on IHostApplicationLifetime.ApplicationStarted (after every
hosted service, incl. an Akka.Hosting AkkaHostedService registered later), or
in StartAsync when no lifetime is registered.
- tests: EagerActorStartTests (generic host in both registration orders with a
no-actor-before-start control; bare container) - all fail without the fix
- docs: README, CHANGELOG
- code-reviews: Secrets-010 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ExpandConfigurationAsync wrote through config[key] = ..., which only sets the
providers' in-memory Data; a file provider's reload replaces it, so editing
appsettings.json on a running service reverted every expanded key to its
${secret:...} placeholder.
- ConfigurationManager: expanded values are pinned in an internal overlay
source appended last; its provider's Load() is a no-op and Build always
returns the same provider, so reloads (and source rebuilds) keep the pins
- any other IConfigurationRoot: in-place write as before, re-applied from the
root's reload token
- all references resolve before anything is written (a fail-closed miss no
longer leaves the config half expanded)
- tests: SecretExpansionReloadTests (real JSON file + Reload(), options
monitor, plain root); three fail without the fix
- docs: README reload note, CHANGELOG
- code-reviews: Secrets-003 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SQL-Server extensions called the 3-arg AddZbSecrets, and the store-path
validator captured the content root of whichever call registered it first.
In hub mode no registration order applied the pre-host under-content-root
guard: extension-first dropped the app's 4-arg root, core-first lost the
replicator (#30). That is the 0.6.0 store-path hole, reopened.
- AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication /
AddZbSecretsAkkaReplication gain a trailing optional contentRootPath,
forwarded to the 4-arg AddZbSecrets
- the root now lives in one SecretsContentRoot instance holder per service
collection; the last explicit root wins, null never erases one; the
IHostEnvironment fallback is unchanged
- tests: ContentRootTests (hub mode, three compositions, artifact asserted
before the throw, positive control), last-explicit-wins theory in
SecretsStorePathRulesTests, Akka forwarding test
- code-reviews: Secrets-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication (and the Akka
extension, same defect class) TryAdd-ed their ISecretStore, ISecretsStoreMigrator
and ISecretReplicator ahead of their internal AddZbSecrets call. An application
that had already called AddZbSecrets therefore silently kept the SQLite store
(shared-store mode) or NoOpSecretReplicator (hub/Akka modes).
They now call AddZbSecrets first and RemoveAll + AddSingleton their own
registrations; a later AddZbSecrets only TryAdds, so either order yields the
same graph with exactly one ISecretStore registration.
- tests: RegistrationOrderTests (SqlServer + Akka) build and resolve the real
container in both orders; core-first fails without the fix
- Directory.Build.props 0.6.5 -> 0.6.6 (NOT packed or published)
- code-reviews: Secrets-001 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
A browse-subtree scope such as Plant/LineA/* filtered out Plant and
Plant/LineA, so scoped lazy browse from the root was empty and
DiscoverHierarchy returned orphans. Out-of-scope ancestors of an allowed
object are now returned as navigation-only containers: attributes stripped,
child_has_children = true, and only their children on the path to an allowed
object are listed (siblings stay hidden). DiscoverHierarchy reports them in
GalaxyHierarchyQueryResult.NavigationAncestorCount; WatchDeployEvents
subtracts it so its scoped counts stay allowed-objects-only.
- tests: ScopedAncestorBrowseTests; deploy-event scope test updated
- IGalaxyBrowseScopeProvider documents the semantics; CLAUDE.md 0.2.1 notes
- code-reviews: GalaxyRepository-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Both projectors memoized one full result list per filter signature with no
limit, and the signature is client-controlled (tag_name_glob '*', '**', ...),
so distinct globs retained memory until the next Galaxy deploy. A bounded FIFO
ProjectionMemo<T> now backs both memos; an evicted signature is recomputed.
- tests: ProjectionMemoBoundTests (500 distinct globs per projector, re-query
after eviction)
- Directory.Build.props 0.2.0 -> 0.2.1 (NOT packed or published)
- code-reviews: GalaxyRepository-001 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The trailing disabled="@Busy" overrode the splatted attribute, so a caller's
disabled was dropped whenever Busy was false and the button stayed clickable.
The effective state is now Busy || caller disabled (Blazor splat semantics).
- tests: caller disabled (true/"disabled"/""), false, and re-render toggle
- Directory.Build.props 0.5.0 -> 0.5.1 (NOT packed or published)
- code-reviews: Theme-007 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
LeaderResolver searched every instance of the application, so co-located
groups (QA: ScadaBridge central-a and site-a both on vq03) got the
first-listed group's instance name on their leader chip. Resolution is now
scoped to the group's members; an unmatched address keeps its raw form.
- tests: QA co-located central/site layout; cross-group host not borrowed
- CLAUDE.md: correct the leader-resolution / 127.0.0.1 guidance
- code-reviews: Overview-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ZbHealthReportClient caught only HttpRequestException/OperationCanceledException,
so an unknown Content-Type charset (InvalidOperationException from
ReadAsStringAsync) escaped the probe, faulted Task.WhenAll and the poller
BackgroundService, and StopHost took the dashboard down.
- client: final catch-all returns an Unreachable result for that instance
- poller: per-target Guarded probe + catch-and-log per sweep in ExecuteAsync
- tests: unknown charset (both tiers), arbitrary handler exception, sweep with
one bad instance, and the BackgroundService survival loop
- code-reviews: Overview-001 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
First reviews of Secrets, LocalDb, GalaxyRepository and Overview, and
2026-11-09 re-reviews of Auth, Health, Theme, Telemetry, Audit and
Configuration, all at 91fb2b4. 49 new open findings, each with a P1/P2/P3
priority and a failure scenario: 1 P1, 13 P2, 35 P3. README regenerated;
REVIEW-PROCESS, template and regen script now cover all ten libraries and
the Priority field.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SPEC 2.4 named OtOpcUa and MxGateway as users; neither is (verified against every
app's default branch; MxGateway worker IPC is named pipes, OtOpcUa deliberately
does not gate readiness on gateway channels). Keep the check (removal is a breaking
API change; small, tested building block), mark it as no-current-consumers in SPEC
2.4/6/7, both READMEs, the shared-contract doc and GAPS.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The two Unix-permission tests called Skip.IfNot from a plain [Fact] (xunit v2 +
Xunit.SkippableFact) so they failed on Windows; now [SkippableFact].
Resolver_Works_EndToEnd_ThroughDi calls SqliteConnection.ClearAllPools() before
deleting its temp DB.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
10.0.7's pool can hand one handle to two users (efcore#39009). 10.0.13 floors
SQLitePCLRaw 2.1.13, so the transitive lib.e_sqlite3 pin moves 2.1.12 -> 2.1.13
(the old pin is a NU1109 downgrade error). NuGetAudit clean; all suites green.
Versions bumped, NOT published: Secrets 0.6.5, LocalDb 0.3.5, Auth 0.4.3.
GalaxyRepository and Overview do not use Microsoft.Data.Sqlite.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audited the registration and mapping code of each app at its origin default branch: OtOpcUa
c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health
0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active
by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table;
each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first
current-state doc.
Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is
merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad.
Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays
deferred (shared rule already used, only the interface is local).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Several setups migrating one SQLite secret store at once (mxaccessgw's parallel test hosts on the
per-process store) failed intermittently on Windows with "cannot start a transaction within a
transaction" from SqliteSecretsStoreMigrator.MigrateAsync. That message means the BEGIN ran on a
handle that was already in a transaction: the pinned Microsoft.Data.Sqlite 10.0.7 can hand one
pooled handle to two owners (dotnet/efcore#39009, fixed in 10.0.13) and no 10.0.x refuses to
re-pool a handle whose rollback failed (dotnet/efcore#39082).
The migrator now serialises migrations of one store per process behind a gate keyed on the
normalised path, runs each attempt on a dedicated unpooled connection inside BEGIN IMMEDIATE, and
retries SQLITE_BUSY/SQLITE_LOCKED (another process past the busy timeout) with equal-jitter
exponential backoff. Store operations and the Microsoft.Data.Sqlite pin are unchanged.
Tests: concurrent-init (16 hosts x 20 rounds on fresh and existing stores, two path spellings,
16 AddZbSecrets containers running the pre-host migrator + hosted service), gate sharing/waiting,
BUSY retry past the 5 s busy timeout, transient-code classification.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
All 22 package projects named gitea.dohertylan.com/dohertj2/zb-mom-ww-<lib>,
which return 404; the libraries live as plain directories in scadaproj.
Takes effect on each library's next published version.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Verified 2026-11-09 against each app's origin default branch pins
(OtOpcUa origin/master c904472a, mxaccessgw origin/main de86f8b8,
ScadaBridge origin/main 7153a197, HistorianGateway origin/main 4d6a9ad)
and the Gitea NuGet feed:
- Audit README: consumer matrix no longer says "adoption deferred" (all four
apps pin 0.1.0, DEEP-adopted 2026-06-02; HistorianGateway row added);
dead ~/Desktop/scadaproj-audit paths fixed.
- Configuration CLAUDE.md: drop "not yet pushed to remotes"; all four pin 0.1.0.
- Telemetry README/CLAUDE.md: all four apps call AddZbTelemetry and pin 0.1.0;
HistorianGateway added to the consumer matrix.
- Secrets CLAUDE.md: lead now says family 0.6.3 on the feed, all four apps pin
0.6.2, none pins 0.6.3 or Replicator.Grpc.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds an additive AddZbSerilog(builder, configure, configureSerilog) overload
taking ZbSerilogOptions:
- MinimumLevelGovernor: Func<IConfiguration, LogEventLevel?> applied with
MinimumLevel.Is AFTER ReadFrom.Configuration, so an app-owned key (e.g.
ScadaBridge:Logging:MinimumLevel) outranks Serilog:MinimumLevel; null defers
to config; Serilog:MinimumLevel:Override:* entries still apply.
- ConfigureLogger: runs last, for app filters/enrichers/sinks (ScadaBridge's
standby active-node exclusion).
The two-argument overload is unchanged. 8 new tests (4 fail with the hook
neutralized). Telemetry + Telemetry.Serilog 0.1.0 -> 0.2.0, not published.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Flat 100*attempt backoff let concurrent boots retry in lockstep (Auth's twin loop
exhausted retries ~1/60 eight-way boots). Backoff is now exponential with equal
jitter, via an injectable Random for deterministic tests.
Refs scadaproj#18
Adds the family backlog standard to the root index (open work = Gitea issues,
shared label set, BTDB excluded) and converts components/*/GAPS, hosted-lib
CLAUDE.md notes, code-reviews, infra/glauth and plan docs to banner + issue
pointers (scadaproj #5-#19, ScadaBridge#79).
- Request UserNameAttribute in the search (de-duplicated, case-insensitive) and return its
first value as LdapAuthResult.Username, falling back to the typed name (I2).
- The empty-password guard now applies in ServiceAccount mode too: AD treats an empty-password
simple bind as an unauthenticated bind and reports success (I3).
- Restore the dropped explanatory comments in CompleteWithServiceAccount / SearchUser.
Review finding (Fix round 1, Task 1): the property getters read their
backing fields without synchronization while RecordSuccess/RecordFailure
write them under _gate, so a concurrent read could observe a torn
DateTimeOffset or a mix of fields from two different writes. Converted the
four auto-properties to explicit properties over private fields, each
getter taking the same lock the writers use. Public interface unchanged.
GrpcSecretSyncService's background sweep now records outcomes on a new
SecretsHubSyncStatus (registered singleton, exposed as public
ISecretsHubSyncStatus) and logs three new events on the outcome-recording
path: EventId 4 (Debug, every in-sync sweep), EventId 5 (Information, the
same outcome throttled to at most once per the new
SecretsGrpcHubClientOptions.InSyncLogInterval, default 1h), and EventId 6
(Information, the first sweep to succeed after one or more failures).
EventIds 1-3 are unchanged; the raw internal SweepAsync still throws on
failure exactly as before. Not yet published to the Gitea feed and
ScadaBridge is not re-pinned — that is a separate, later step.
Replaces the 2026-09-11 remote-exec hold: WinRM exec is allowed only for the on-box steps of a
user-requested deploy; ad-hoc diagnostics/version checks use SMB, HTTP or SQL. SMB file copies and
direct SQL logins remain always allowed. Scope rules live in the winrm skill's Scope section.
CLAUDE.md: ScadaBridge row records main @ 17864d1a (records only, code tip
2b9a7c3c) deployed on vd03 and the vq03/vq04 pair; Theme/Auth/LocalDb rows
lose their "not yet deployed" caveats (QA pair now runs LocalDb replication,
connected); tier note gains the deployed-state line and the winrm access path.
deployment.md: §2.1 access rows replaced by the winrm skill (WinRM 5985 +
SMB 445; servecli/psbridge retired), §7.1 port map gains 5985/445 and marks
6150 retired, live-state table updated (OtOpcUa active flipped to node-a),
new "Deployed builds on QA" paragraph, §9 deployed-state summary.
Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
Row in the sister-project table, a BTDB-edges subsection (Galaxy write contract, ScadaBridge inbound-API SAPID lookup on db-connection BTDB, Historian CLR/INSQL co-location), break-surface note, Windows-only build commands, and the DARS bullet now points at the unversioned WW_ARCH/BatchTracker_DB fork. Full schema/contract notes and the 2026-09-03 gap analysis live in .claude/skills/sister-project-history/btdb.md.
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
.gitignore ignored all of .claude/, so the per-project history files the root index points at were never committed; the pushed index had lost that history. Un-ignore .claude/skills/ (settings.local.json and worktrees stay ignored) and add the three files.
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
SnapshotStreamer.SendAsync stamped as_of_seq = MAX(seq), which reads 0 once ack-pruning has
emptied a converged sender's oplog. A receiver whose watermark genuinely is 0 (a rebuilt node —
ScadaBridge's site-a-a since its 2026-08-15 rebuild) was healed by the snapshot but never lifted
by it (the receiver's watermark update is correctly MAX-guarded), so the 0.1.2 empty-oplog
horizon rule saw 0 < last_acked_seq as a gap on every subsequent handshake and re-streamed the
whole database on every reconnect until the sender wrote again. Observed live: the same 127k-row
full snapshot on two consecutive restarts of the pair.
Fix: as_of_seq = max(MAX(seq), last_acked_seq). Every delta pruned below last_acked_seq is
already folded into the rows the snapshot carries, so that is the floor it may claim; the
receiver's MAX guard then lifts the watermark and the next handshake finds no gap. Sender-side
only; wire-compatible; no schema change.
Test WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot fails on the
unfixed code exactly as the rig does (watermark 0, expected 3) and passes with the fix: the
second episode exchanges only a delta, and a capturing logger on the snapshot streamer proves
no snapshot was sent. Suite 216/216 on 8 of 9 runs; one unnamed single-test failure on the first
run did not recur in eight further runs and could not be attributed.
Three review follow-ups on 122940c. Still 0.3.3 — unpublished, so no further
version bump. No behaviour change to any succeeding path.
- The per-entry poison handler's recovery work (ROLLBACK TO + the dead-letter
INSERT) was unguarded: a wedged connection, an I/O error, or a dead-letter
INSERT that faults would propagate the SECOND exception and silently discard
the FIRST — the one that says why the entry was poison at all. The pair is now
wrapped; a recovery failure throws an AggregateException whose message names
the entry (table + seq) and the recovery fault, and whose InnerException is the
ORIGINAL entry fault, with the recovery fault carried alongside it at
InnerExceptions[1] with its own stack. The outcome is unchanged and still
fail-closed: nothing commits, and disposing the batch transaction unwinds it.
OperationCanceledException is deliberately excluded from the wrap so a
shutdown still surfaces as a cancellation rather than as a fault. No upstream
caller catches SqliteException from the apply path (checked), so widening the
escaping type is safe.
Injected cheaply after all: a RAISE(ABORT) trigger on __localdb_dead_letter,
stacked on the existing row-version poison trigger, produces exactly the
two-fault case with no production test seam. The test asserts both causes are
carried AND that the batch committed nothing — the entry that applied cleanly
before the poison one is gone and the peer watermark never moved.
- Register_SelfReferencingCascade_Throws pins the README's "or from itself"
claim: a table that is its own cascading parent (parent_id REFERENCES
nodes(id) ON DELETE CASCADE) is refused. The guard needed no special case —
it reads every foreign key in the database — but the promise is now tested.
- Comment on why EnterEntryAsync sits outside the try: a SAVEPOINT that failed
opened nothing, so a handler ROLLBACK TO would name a savepoint that does not
exist and fault in turn.
Both new tests were run against 122940c first and fail there: the self-reference
case throws nothing, and the stacked-fault case escapes as a bare
SqliteException("dead-letter write failed") with the original cause gone.
Suite: 215/215 green, build 0 warnings / 0 errors.
Two latent apply-path hazards from the ScadaBridge adversarial architecture
review. Neither has a live trigger in any current consumer — ScadaBridge's ten
replicated tables carry no foreign keys at all (verified against the running rig's
site-localdb.db) and OtOpcUa's LocalDb tables carry none either — but both are the
kind that surface as silent divergence rather than as an error, so they are closed
before a consumer's schema grows into them.
Wire-compatible: no proto change, no lib_schema_version bump, no on-disk schema
change. No consumer code change.
- RegisterReplicated now refuses a table that is the PARENT of a foreign key with
a row-mutating ON DELETE action (CASCADE / SET NULL / SET DEFAULT), naming the
offending child column and action. The apply path writes remote rows with
INSERT OR REPLACE, whose PK-conflict resolution DELETES the existing row first;
connections run PRAGMA foreign_keys=ON, so that implicit delete fires the child's
ON DELETE action while the capture guard is raised. The child rows would change
on the applying node only, produce no oplog entry, and diverge the pair with
nothing able to detect it — LWW compares only rows it was told about. RESTRICT /
NO ACTION stay allowed: they abort the statement instead of mutating rows behind
capture's back, which the apply path already handles as per-entry poison. The
check inspects every foreign key in the database and runs at registration;
SQLite cannot add a foreign key to an existing table, so a child table created
later is caught at the next startup's registration, not at the DDL. Documented
on ILocalDb.RegisterReplicated and in the README's limitations list.
- An entry's row write and its ledger write are now atomic with each other. They
are two statements and SQLite aborts only the failing one, so a statement-level
fault between them — the case the per-entry poison catch exists for — committed
a half-applied entry: a tombstone whose table row was gone while
__localdb_row_version still called it live (SnapshotStreamer can only warn and
skip that, so the peer never learns the row died), or an upsert whose new row
carried the OLD version and therefore lost the next comparison it should have
won. The pair now runs inside a per-entry SAVEPOINT, rolled back (and released,
so nothing nests) before dead-lettering, leaving both writes or neither while
the rest of the batch proceeds and commits.
Every new test was run against the unfixed code first and fails there with exactly
the divergence shape described: the tombstone case reads back null instead of the
live row, the upsert case reads back the remote row against an unchanged ledger,
and the registration cases throw nothing.
Suite: 213/213 green, build 0 warnings / 0 errors.
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.
RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.
New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).
BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)
Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
Secrets/push.sh, Theme/pack.sh+push.sh, Audit/pack.sh+push.sh and Auth/pack.sh+push.sh
all carried the CWD-pack bug LocalDb's push.sh was cured of in 0038d46: a bare
'dotnet pack -c Release -o ./artifacts' packs whatever solution sits in the CALLER's
working directory, and the push scripts then shove the caller's ./artifacts/*.nupkg
onto the live Gitea feed (the incident that briefly published 25 unrelated packages).
Each script now cds to its own repo root, names its .slnx explicitly, writes to an
absolute artifacts path, and (push.sh) pushes an anchored glob. Verified by bash -n on
all seven plus a positive control: Audit's pack.sh run from a foreign CWD packed only
ZB.MOM.WW.Audit into its own artifacts dir and left the caller's directory untouched.
Push scripts deliberately NOT executed (they publish); pack half verified only.
Closes ScadaBridge deferred-work register row 42 (cross-repo).
dotnet pack with no project argument packs whatever solution sits in the caller's
CWD, so invoking this script by absolute path from another repo's checkout packed
and published 25 unrelated packages to the live Gitea feed under its name. Now it
cds to its own repo root and names the solution and an absolute output path.
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.
Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.
Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.
For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.
Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.
Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.
README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
The umbrella LocalDb row and the pack/test quick-reference now name 0.2.1 (188 tests,
3 nupkgs on the feed). Records both findings, the snapshot-out-of-handshake change that
bounding the inbox required, and the deferred per-table needs_snapshot. Consumer pins are
unchanged: ScadaBridge still pins 0.2.0, OtOpcUa still pins 0.1.3.
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.
F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.
F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.
Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).
Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.
188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.
Version 0.2.1.
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.
DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.
RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.
O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.
Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.
LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.
Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.
184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.
Version 0.2.0.
Two local glauth containers (zb-ldap-primary :3893 / zb-ldap-backup :3894)
simulating the NAM two-DC topology, so the primary can be killed during a
drill without touching the shared 10.100.0.35 instance. Drill recipe +
verified-PASS record (HistorianGateway, 2026-08-13) in the README.
deployment.md:
- §1/§2.1: add wonder-app-vq03, wonder-app-vq04 and wonder-sql-vq03 with IPs and
per-host roles; new §7.1 QA port map + live active/standby state.
- QA is a real 2-node HA pair (ScadaBridge central+site pairs, OtOpcUa pair) where
DEV is a single all-in-one box, so it is the only environment here that exercises
failover and leader election. MxGateway/HistorianGateway/Overview run on vq03 only.
- Every QA fact is marked scan (port probe) or dash (read off the QA Overview
dashboard's live polling). No QA config file was read; connection strings and
service names stay unverified.
- Correct the tier: vd03 is DEV, not production (vd=dev, vq=qa, per
mes-delmia-integration-api.md §1.1). Its ASPNETCORE_ENVIRONMENT really is
"Production" -- a .NET overlay name, orthogonal to the tier, and the likely source
of the original error. No production box has been identified.
- Record that QA requires Basic auth on /management where DEV does not, so DEV-only
procedures relying on unauthenticated management calls will not work on QA.
- Record that inbound API keys never cross environments (per-stack peppered-HMAC,
excluded from config bundles) plus the 401-vs-400 probe, with the caveat that auth
runs before route resolution so a 401 needs a positive control.
- Withdraw the SSH-on-2222 claim made earlier in this same pass: all three wonder
boxes accept TCP on 2222 and send no SSH banner, so servecli is not serving SSH and
no shell access was available. Port scan cannot establish a service; banner grab can.
CLAUDE.md:
- Overview: record the second instance on wonder-app-vq03:5320, and that its real-FQDN
registry is correct rather than a divergence -- the 127.0.0.1 rule is specific to
vd03 co-locating every instance on one host.
- Add the DEV-vs-prod note pointing at deployment.md §2.1/§7.1.
CLAUDE.md: feed now carries 0.2.1; record the referral defect and that every
app is still exposed until it bumps off 0.1.5 (a functional fix, not
housekeeping), plus the new tools/LdapProbe.
ldap_setup.md: findings from probing the live NAM directory --
- LDAPS 636, StartTLS 389 and plain 389 all connect on both DCs. The only TLS
chain error is UntrustedRoot, so installing zimmer-rootca is sufficient; the
RSA 1024-bit key is rejected only by OpenSSL at SECLEVEL=2, i.e. Debian .NET
containers, not by macOS or Windows hosts.
- The UG_WW_* Wonderware group convention, 39 groups enumerated directory-wide.
This partly overturns this doc's earlier "no group->role convention exists"
claim: that was true of Ignition, which was the only source audited, and
false of the domain. Three shapes with a consistent role suffix (ADMIN, RO,
OPERATOR, ENGINEER, SUPERVISOR, TESTER) across environment and site axes.
- A proposed mapping onto CANONICAL-ROLES, with the three gaps left open
rather than guessed: SUPERVISOR/TESTER have no canonical equivalent, nothing
maps to Deployer, and the DEV/QA/PROD axis is orthogonal to role so it must
ride in the per-project scope payload or the role set triples.
- The referral defect and its fix.
Diagnostic for pinning down the LDAP settings the four ZB.MOM.WW apps need
against the Zimmer NAM domain, and for answering which AD groups can be mapped
to app roles. Found the referral defect fixed in 0.2.1.
Two independent halves in one run:
Part A drives the REAL ZB.MOM.WW.Auth.LdapAuthService -- same bind-then-search-
then-bind, same options validator, same failure enum -- so its verdict IS what
the apps produce rather than an approximation of it.
Part B asks the directory what the library cannot, which is where the group
answer comes from: full group DNs, nested groups via
LDAP_MATCHING_RULE_IN_CHAIN, the primary group (AD omits it from memberOf, so
it can never be a role group), security vs distribution type, decoded
userAccountControl, referral counts, and the TLS chain with key sizes.
--list-groups '<pat>' enumerates every matching group in the directory, which
a membership dump structurally cannot do -- it only ever shows one account's
memberships.
Passwords are prompted for and never echoed; they cannot be passed as
arguments, which would put them in shell history. --self-bind lets the user
under test stand in for the service account, so group discovery does not
require the svc_igntnapp secret.
Not packable, outside the Auth solution and its central package management,
referencing the Auth projects directly so it tracks current source.
Active Directory answers any search based at a domain root with continuation
references (CN=Configuration, DC=ForestDnsZones, ...). With referral following
disabled -- the default, and what we want, since chasing one would re-bind
somewhere we never authenticated -- Novell surfaces each by throwing
LdapReferralException from Next().
That escaped NovellLdapConnection.Search, and LdapAuthService maps a failed
search to ServiceAccountBindFailed. So every login against
SearchBase = DC=nam,DC=zimmer,DC=com failed system-side on BOTH DCs in ~546 ms
while the directory, the credentials and the service account were all healthy.
Identical on each endpoint, so failover could not help either.
A continuation reference is a pointer to another naming context, never a
statement that this search failed, so the loop now skips them and continues --
returning exactly the entries in the naming context we bound to. Extracted as
ReadEntries(ILdapSearchResults) so the handling is unit-testable through the
Novell seam without a live directory. A 1000-referral cap guards a client that
throws without advancing, which would otherwise spin forever inside a
fail-closed auth path.
Non-referral faults still propagate untouched: swallowing those would turn
"the directory is broken" into "no such user", which is the wrong answer to
give about someone's credentials.
7 tests, 6 verified red against the pre-fix code; the seventh is the
non-referral control that must stay green either way. Suite 257/0, 2 skipped.
Live-proven against the ZB NAM domain root before and after.
Asked whether the gateway's new session row also covered HistorianGateway.
It did not, and could not: HistorianGateway publishes its pool depths on a
different TIER with a different KEY SET.
mxaccessgw active tier total / ready / faulted one check
historiangw READY tier warmSessions / pool / … TWO checks
The session row is located by the total/ready/faulted triple, so it correctly
ignored these rather than half-matching them. This adds a sibling row instead
of unifying the two behind a shared shape, because there is no such shape -
inventing one would mean rendering a contract neither app publishes.
No poller change was needed: the ready tier's body is already retained, so
this is entirely a rendering change.
Labelled "Sessions pooled", deliberately not "connections". warmSessions is
pool DEPTH - warm pre-authenticated sessions held ready - and sessions are
leased on demand, so the event pool legitimately sits at 0 until something
reads events. Calling that a connection count would make a normal idle pool
read as an outage, so 0 is shown plainly and the row goes red only on
consecutiveFactoryFailures, which is an actual fault.
Pools render read-then-event rather than in dictionary order, so the line is
stable across polls and the pool an operator cares about comes first.
188 tests (183 + 5): ordering, that an idle pool is not painted red, that a
failing factory is, a positive control on the omission assertion, and one
pinning that neither app's row can render the other's data.
The gateway publishes its live MXAccess session counts, and the dashboard
could not reach them. Four things stood between the data and the card, and
only the last one is cosmetic:
The active tier was probed only for instances declared HasActiveRole. A
standalone gateway has no active/standby role, so its active tier was never
requested at all. The tempting fix is to declare mxgw HasActiveRole, and it
is wrong: that flag also paints an Active/Standby badge, so it would buy the
data by asserting a role that does not exist. The probe is now unconditional
for any reachable instance while ObserveActive stays role-gated, separating
"what does this tier report" from "am I the active half".
The active probe's parsed body was then discarded - only ready.Report was
kept - so InstanceSnapshot gains ActiveReport. The two tiers stay separate
rather than merged, because a check name is only unique within a tier.
The card's one data-rendering path, ClusterDataLine, is located by the Akka
leader/activeNode keys and would ignore session data even once present. The
new row follows that helper's own precedent and locates the check by the data
it carries - the total/ready/faulted triple - not by name, requiring all three
so a future check publishing a lone "total" cannot be mistaken for it.
Rendered on the card FACE, not inside the collapsed check list: a count behind
a disclosure triangle is one nobody reads. Zero is shown rather than hidden,
since a gateway holding no sessions is the interesting case and suppressing it
would make the row appear only when the news is good.
Cost, stated plainly: every reachable instance now makes one extra HTTP
request per poll. The unreachable-instance skip is untouched, so the
worst-case sweep duration - which is set by timeouts, not by healthy
round-trips - is unchanged. Four cadence tests were counting raw requests as a
proxy for "instances polled"; they now count ready-tier probes, which is what
they always meant and is robust to tier changes.
The Wonder registry comment claiming the active tier ran an empty check set
was true when written and stopped being true when the gateway added the
check; it now warns against the HasActiveRole shortcut instead.
183 tests pass (177 + 6), including a positive control on the absence
assertion so "row not rendered" cannot pass vacuously.
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.
A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.
Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
Raised by the mxaccessgw session, which went back to its own tests after I reported a
vacuous assertion in mine and found the same class one step further from the fuse.
The rejecting test asserts no store file is left at a rejected path. That claim is only
meaningful if something proves the write path is live at all: "no file was created" is
indistinguishable from "nothing in this container ever creates files". The positive
control was supposed to establish that and did not -- it asserted Assert.NotNull on a
resolved SecretsSqliteConnectionFactory, which is close to a tautology once no exception
escaped, and it never ran the migration at all. So if the pre-host container ever stopped
touching the store, the rejecting test would have gone quietly vacuous while the positive
control kept passing. Neither would have failed.
The positive control now runs SqliteSecretsStoreMigrator.MigrateAsync and asserts the
database IS created. The pair is mutually checking: the migration writes the file when
allowed and does not when rejected, and neither assertion stands on its own.
Verified the same way as the last one, by simulating the failure rather than reasoning
about it: replacing the migration with a no-op makes the positive control fail with "the
accepted path must actually get a store file". Reverted after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Two defects in the 0.6.1 regression test, the first raised by the mxaccessgw session
from its own adoption, the second exposed by acting on it.
Assertion order. The test asserted the throw before the artifact. That hides the
stronger finding: a regression that stops the throw but still creates the store fails
with "no exception was thrown" and never mentions the database sitting at the rejected
path -- which IS the defect. Now captures with Record.ExceptionAsync, asserts no file
created first, and asserts the exception type last. Notable because the defect under
test was itself "a boot outcome was observed instead of the artifact being probed", so
closing it with an outcome-first assertion would have repeated the mistake.
Vacuous assertions. Simulating the regression -- replacing the 4-argument call with the
3-argument one, which still compiles, exactly as the call-site comments warn -- showed
the test failing on the exception assertion rather than the file assertion. No file had
been created in EITHER case: the test resolved SecretsSqliteConnectionFactory, whose
constructor never touches the filesystem. The File.Exists assertions could not fail. The
test now resolves SqliteSecretsStoreMigrator and runs MigrateAsync, which is what a
pre-host expander actually does and what actually creates the database.
Verified by simulating the regression again: the test now fails with "a rejected path
must not be left holding a store file" rather than "no exception was thrown". Reverted
after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.
Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).
Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.
The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.
BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.
Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.
Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.
Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.
Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
This repo hosts one application besides its shared libraries, and the index did
not mention it at all. Adds what it is, that its registry ships empty by design,
and the two constraints that bit during the first real deployment: publish
self-contained (wonder has no .NET 10 runtime), and do not put co-located
instances on a shared host name, because leader-to-instance resolution matches on
host only and the first match wins the chip for every group.
Live on the box, the ScadaBridge *site* card claimed "Leader · central". Central
and site are separate Akka clusters, so that name was simply wrong.
Cause is in the registry, not the app. ResolveToInstanceName matches a reported
Akka leader address to an instance on HOST ONLY - correct by design, because an
app's Akka remoting port and its HTTP port differ, so comparing ports would never
match. It assumes what holds on the docker rig and not here: distinct hosts per
instance. All five instances are on one box, and the scan is fleet-wide in
registry order, so with every BaseUrl on "localhost" the first matching entry won
the chip for every group.
No host value makes this resolve CORRECTLY, because the instances really are
co-located. So make it never match instead: ScadaBridge's Akka addresses use host
"localhost" and OtOpcUa's uses the FQDN, and "127.0.0.1" is string-unequal to
both. Every chip now falls back to the raw Akka address - unambiguous and always
true. A raw address beats a confident wrong name.
Using the FQDN for OtOpcUa's BaseUrl would have resolved its chip correctly and
was rejected: that name resolves to a link-local IPv6 on this box, and routing
the real health probe through it to improve a label trades the primary function
for a cosmetic one.
ManagementUrl values are unchanged - they are followed by the operator's browser
from another machine and must keep the box address.
Adds appsettings.Wonder.json for the first deployment of this dashboard to a
real box. Named "Wonder" rather than "Production", mirroring the existing
Docker overlay: this is ONE deployment's fleet, not a claim about what every
production deployment looks like.
Every port was established by probing the running box, not inferred from the
docker rig - and the two disagree, so inference would have been wrong:
OtOpcUa :9000 (AdminUI on the same listener)
ScadaBridge Central :8085 (rig serves this on :5000)
ScadaBridge Site :8084 (no UI - / is 404, so the card carries no link)
MxAccessGateway :5130 (:5120 is gRPC)
HistorianGateway :5220 (:5221 is gRPC h2c)
:8083 and :8086 answer 400 to an HTTP/1.1 GET - h2c gRPC - and are excluded.
Both gateways are HasActiveRole:false. Their /health/active does return 200,
but only because the active tier runs an EMPTY check set when no Active-tagged
check is registered, so an Active badge would assert a role neither has.
Central and site are separate groups because they are separate Akka clusters;
one shared group would manufacture a leader disagreement.
BaseUrl is localhost (probed by the service, on the box) while ManagementUrl
carries the box address (followed by the operator's browser, elsewhere).
Serilog's file path is absolute and outside the Server directory an upgrade
replaces - a relative path resolves against the service AppDirectory, so every
upgrade would silently eat the log history. Same class of defect that cost
HistorianGateway its secrets store on a swap.
Boot-verified locally under ASPNETCORE_ENVIRONMENT=Wonder: registry passes
OverviewOptionsValidator and the host starts.
Family alignment ahead of the first wonder-app-vd03 deploy. Both bumps are
inert for this app, established rather than assumed:
- Theme 0.3.1 -> 0.4.1 is two commits: the upstreamed Bootstrap .btn sizing
block and the rail-btn-block modifier. Overview answers NO to both of the
questions the kit README says to ask - it declares no .btn sizing of its own
and has ZERO call sites carrying a bare `btn` token (its single button is
class="rail-btn"), so the new .btn rule matches nothing here. The .rail-btn
refactor that introduced --rail-btn-inset resolves to the same 0.6rem the
literal margin used, so the one call site is unchanged. Same answer as
HistorianGateway. Note a naive \bbtn\b grep is a FALSE POSITIVE on
"rail-btn" - the hyphen is a word boundary; match a whitespace-delimited
token instead.
- Health 0.2.0 -> 0.3.0 is a pure version-number alignment: the core
ZB.MOM.WW.Health package is byte-identical between the two, with the entire
delta confined to ZB.MOM.WW.Health.Akka (ActiveNodeHealthCheck,
ClusterActiveNode, AkkaActiveNodeGate), which this app does not reference.
Verified with a positive control on the diff path. The 0.2.1 active-tier
503 change therefore cannot reach this app's own endpoints; it registers no
Active-tagged check, so /health/active runs an empty set and stays 200.
Overview consumes the 0.3.0 `activeNode` data key over the WIRE from other
apps' health JSON, which is independent of the package it compiles against -
LeaderResolver already ranked it (344bdc4) while pinned to 0.2.0.
Release build 0 warnings; 177/177 tests pass. All remaining outdated packages
are test-only (bunit, coverlet, test SDK); nothing the app ships is behind.
A commit that removed a modal's self-closing behaviour shipped two comments in the
same file still describing it as self-closing — review checks whether the code is
right, not whether the surrounding prose survived. And the property that predicts
which modals break on refresh is 'holds no state of its own' (pure projection of a
list), not the presence of a detail field.
Readers take 'what I could not establish' as evidence the rest was held to that
standard. Write the claim the evidence licenses, not the broader one; and when two
items in your own not-established list contradict each other, flag it — that
contradiction is the strongest lead you have, not another unknown.
A bare 32-hex key id matches none of the six credential keywords (verified). Added
an entropy scan and a PEM check as generic complements, marked the prefix list
repo-specific, and made the primary rule explicit: a gate that mints anything must
grep the commit for the literal values it produced.
A filename audit finds the wrong-file problem; it cannot find a credential pasted
into a legitimately-tracked doc. Both checks are cheap and catch disjoint failures.
Verified: .claude/ is gitignored in OtOpcUa and HistorianGateway, NOT in mxgw or
ScadaBridge — in those two, enumerate-first discipline is the only guard.
A sweep's landing commit is where unrelated local state gets swept in. One repo's
'78 modified' was 76 modified + 2 untracked, one of which was .claude/ holding
settings.local.json — a literal 'git add -A' would have put permission config into
a tracked commit feeding a production deploy, invisible in a 76-file diff.
Two deadlocked investigations both inherited 'file exists => sink constructed =>
no events flowed'; the pinned sink version creates no file until the first emitted
event, so the artifact proved the opposite. And 'a second log wrote in the same
directory' rules out directory-level failure, not per-handle failure — a proof
about the container is not a proof about the member.
HistorianGateway has no local block AND no Bootstrap button call sites (all 15
buttons use the kit's .rail-btn chassis class), so the kit's .btn rule matches
nothing and bumping it changes nothing visible. My previous correction claimed the
opposite. Checking this needs two greps — declaration AND call sites — which come
apart exactly when an app doesn't use Bootstrap buttons.
Three instances in one round (btn-block universality, sweep behavior-preservation,
the mxgw Bootstrap exemption), all the same shape, all caught by the asserter
re-checking rather than by challenge. Fix is a cheap per-member check that runs,
or an honestly-scoped claim when it hasn't.
HistorianGateway has no local .btn sizing in either its committed tree or its
sweep branch (only vendored bootstrap.min.css), so the shipped comment and README
naming all five consumers were wrong. Consequence worth stating: HG's bump to
0.4.0+ is a VISIBLE change (buttons shrink to .85rem), not the no-op cleanup it is
for consumers that already carried the block.
The prior wording claimed the two rules 'cannot drift apart', which is only true
for the documented composition. 0.6rem still appears twice; the fallback confines
a stale copy to the undocumented modifier-alone path. Records why hoisting the
var to :root would be worse rather than better.
Unreleased — behaviourally identical to the published 0.4.1 (browser-verified:
composed <a> and <button> both fill the rail foot exactly, 0.00px short). The
var() carries a 0.6rem fallback because --rail-btn-inset is declared on .rail-btn,
so rail-btn-block used alone would otherwise have an undefined var, invalidating
the whole width declaration at computed-value time and silently reverting to
width:auto — the exact <button> shrink-wrap bug 0.4.1 fixed.
Two lessons from the Theme 0.4.0 rollout across five apps.
Live gate: after a kit-CSS change a normal reload can serve the cached previous
layout.css while the server serves the new one. .btn then computes at
Bootstrap's 16px — the exact signature of a failed upstreaming — so the natural
response is to restore the local rule just deleted, entrenching the duplication
the change removed. Hard-reload before trusting any computed-style probe.
Upstreaming: the kit ships sizing only, so trim the local .btn rather than
deleting it. mxgw's block also carried border-radius/font-weight/white-space,
which a wholesale delete would have dropped silently; three of five apps' blocks
were pure sizing and safe to delete outright, and the difference is invisible
without diffing. Also records why a retained border-radius must stay a literal.
Marks the .btn block as upstreamed in 0.4.0 rather than a candidate.
0.4.0's rail-btn-block used display:block + width:auto, which fills for an <a>
but SHRINK-WRAPS a <button> — a form control does not take the fill-available
behaviour a non-replaced block box gets. Measured in one 197px rail-foot
container: <a> filled to 177.42px (0 short), <button> came up 114px short.
That matters because a rail foot normally holds both element types: Sign In is
an anchor, Sign Out is a form-submit button. So 0.4.0's modifier got one right
and one wrong, and the wrong one is what signed-in operators see — worse than
the width:100% overflow it was meant to replace (~10px of visible clipping).
Fixed with an explicit calc(100% - 1.2rem) + box-sizing:border-box, which
measures 0px short for both element types. Both rejected shorthands and their
measured failures are recorded in the CSS so neither is reintroduced as a
'simplification'.
Found by HistorianGateway while evaluating adoption; it held off adopting
rather than trading one defect for a larger one. No consumer had adopted the
modifier, so nothing shipped broken. The .btn half of 0.4.0 is unaffected and
is already live-verified in three apps.
Build 0 errors, 48/48 tests.
Bootstrap hard-codes --bs-btn-font-size: 1rem, oversized against the kit's
0.9rem body. All five consuming apps had independently written the identical
block into their own stylesheets (ignitionoee app.css, OtOpcUa site.css, mxgw
site.css, HistorianGateway site.css, ScadaBridge site.css), each carrying a
'delete this when the kit ships a .btn rule' comment. This is that release.
Sized by CSS-variable override only, never by redeclaring box properties:
Bootstrap's .btn-group seam/radius machinery reads the same variables, so a
literal font-size at equal specificity flattens .btn-sm into a padding-only
difference. That was a real defect in mxgw's local copy, so the rationale is
recorded in the CSS to stop it being 'simplified' back.
Lands in layout.css, which ThemeHead emits after theme.css and both after
Bootstrap, so the override wins on source order at equal specificity.
Ships in the same 0.4.0 release as rail-btn-block. Consumers get the sizing on
bump (not opt-in, unlike rail-btn-block) and should delete their local copies
in the same change. Build 0 errors, 48/48 tests, packed asset verified to carry
.btn, .btn-sm/.btn-group-sm and rail-btn-block with no literal font-size.
HistorianGateway hand-rolled a .rail-btn override purely to make the rail-foot
sign-out button span the rail; its own comment flagged folding it into the kit.
Adds a composable modifier instead: class="rail-btn rail-btn-block".
Uses display:block + width:auto, NOT width:100%. The hand-rolled overrides set
width:100%, which resolves against the containing block and then adds .rail-btn's
0 0.6rem side margins on top, overflowing the rail by 1.2rem. width:auto fills
the same inset correctly.
Scope note: the app overrides that motivated this do more than full-width (they
also restyle font-size, padding, border, background and hover). Adopting the
modifier removes the width hack; whether to also drop the restyle is each app's
call, since it changes their rail appearance.
Version 0.3.1 -> 0.4.0 (additive). Build 0 errors, 48/48 tests, nupkg verified
to carry the new rule in staticwebassets/css/layout.css. Not yet published.
CLAUDE.md (Secrets row):
- mxgw is NOT Bootstrap-free (libman.json pins bootstrap@5.3.3, App.razor:7
links it ahead of ThemeHead). The scadaproj#2 modal gate had skipped mxgw on
that false premise; gate since run and PASS 2026-08-11.
- scoped-CSS bundle claim scoped to its 2026-07-19 date; OtOpcUa links one now
too, so it is two of four rather than one.
admin_ui_cleanup.md — five rounds now (adds mxgw, HistorianGateway, ScadaBridge):
- 1a-1: test sources are a fifth definition site, checked BEFORE the ghost list.
A class matched only by a test is a selector, never a ghost. Extends to
substring and visible-text assertions, so button labels are load-bearing.
- 1a-4: two inverted button-sizing instances (literal font-size neutering
btn-sm app-wide; a chassis class standing in for the button system).
- 1a-5: determine dark scheme per app; where one exists the step inverts.
- 1a-6 (new): a token override emitted as an inline style on the shell root
defeats every scheme-level :root override. OtOpcUa carries the inert-but-
latent instance.
- 1c: reword nothing before checking test sources; the docs may be the wrong
document, not the page.
- 2: btn-sm fold breaks button.btn-*.btn-sm locators; cell-clip is wrong for
resizable columns; briefs carry the rubric, not just line numbers.
- verification: baseline pre-existing failures by measuring; a suspiciously
fast green is a vacuous pass, not a pass.
- live gate: mandatory bUnit or not; hit-testability over visibility; measure
behaviour where computed values mislead (-webkit-box serializes flow-root);
record what could not be exercised.
- new: exemptions from a family-wide sweep must re-derive their premise.
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bump ZB.MOM.WW.Secrets family 0.4.1 -> 0.5.0 and date the CHANGELOG
entry. All 6 packages (Abstractions, Secrets, Replicator.SqlServer,
Replicator.AkkaDotNet, Replicator.Grpc, Ui) packed and pushed to the
dohertj2-gitea feed; restore-verified from a scratch consumer with a
fresh package cache. Suite 470 passed / 0 failed / 15 skipped.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Review follow-ups: the warning now carries only the two endpoint URIs and the
status code — attaching the exception object would let any real sink render
its ToString(), which for RpcException includes server-authored Status.Detail.
The leak pin test now also asserts no exception rides on the warning, via a
new WarningExceptions capture on RecordingLogger. Plus two cosmetic nits:
unified ThrowIfNull idiom, documented that FallbackEndpoints mutation after
Validate is unsupported.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
SecretsGrpcHubClientOptions gains a getter-only FallbackEndpoints list:
additional hub addresses tried in order when a call fails with a
transport-level RpcException. Validation fails startup on blank/relative
entries and on any duplicate of the primary or another fallback (a
duplicated endpoint silently halves the configured failover).
New FailoverSecretsHubReader composes one GrpcSecretsHubClient per
endpoint: the answering endpoint is sticky (a sweep is GetManifest + N
GetSecrets batches — without stickiness every call after a primary
outage re-pays a full CallDeadline before reaching the healthy
fallback), rotation wraps so a recovered primary is retried when the
stuck-to endpoint fails, only RpcException triggers failover
(cancellation and defects propagate untouched), and when every endpoint
fails the last RpcException is rethrown so the sweep degrades to
last-known-good exactly as today. Failover warnings name only the two
endpoint URIs and the status code — never any credential material.
DI: zero fallbacks keeps the registration graph byte-identical to 0.4.x
(single keyed channel under ChannelServiceKey, plain
GrpcSecretsHubClient); with fallbacks, one container-owned keyed channel
per endpoint (fallback i under "{ChannelServiceKey}:fallback:{i}") wired
into the failover reader. GrpcSecretsHubClient itself is unchanged.
Safe only when every listed endpoint serves the SAME store — for
ScadaBridge, the central pair over one shared SQL Server store; the
options doc and CHANGELOG carry the warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
An internal validateEagerly seam on AddZbSecretsGrpcHub replaces the harness's
hand-copied registration fragment, so the deny-all second-layer tests exercise
the production wiring and cannot drift from it.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the end-to-end layer the unit suites structurally cannot reach: a real
in-process ASP.NET Core host serving SecretsHubService through
AddZbSecretsGrpcHub/MapZbSecretsHub, a follower built by the shipped
AddZbSecretsGrpcHubClient, a real gRPC call between them, and real SQLite
stores on both sides. The class of defect this catches is pieces that are each
individually correct and are never connected to one another — for instance the
interceptor's warn-once, which only holds if the container really hands gRPC
the singleton it registered.
Covered: byte-identical convergence over the wire (every ciphertext/nonce/tag/
wrapped-DEK byte, kek_id, revision, both timestamps INCLUDING their offsets,
across a tombstone, a nulls-everywhere row and an empty-string-description row);
delete propagation; a hub unreachable at follower boot leaving the last-known-
good store intact and converging once it answers; the auth negatives fail-closed
across the wire with one uniform denial detail; the per-request name cap as
ResourceExhausted; and an RpcException from the call invoker escaping the hub
client unchanged, so a transport fault stays distinguishable from a parse fault.
The credential-hygiene assertions are made against logs captured at Trace on
BOTH hosts, so "no token material ever reaches a sink" is asserted at the most
verbose setting a deployment could switch on.
SecretsHubContractPinTests reads the generated descriptor — not the .proto text
— and pins the RPC set to exactly {GetManifest, GetSecrets}, plus unary-only and
the package-qualified service name. The hub is pull-only by construction:
updates originate at central only, and the absence of a write RPC is what
enforces that, not authorization a mis-wired follower may hold more of than
intended. A third RPC silently retires the guarantee, so the failure message
says so.
Microsoft.AspNetCore.TestHost pinned at 10.0.7 — the version ZB.MOM.WW.LocalDb,
OtOpcUa and ScadaBridge already pin, matching the Microsoft.Extensions.* line.
Full suite: 436 passed / 0 failed / 15 skipped, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
A single malformed row or manifest entry poisoned the whole sweep, permanently.
WireToDomain.ToRows/ToManifest materialized eagerly, so one unparseable element
threw out of the fetch callback, past SecretReplicationReconciler's per-batch
handling, into SweepSafelyAsync's whole-sweep catch. The follower's pull set is
derived from the hub's manifest, so the identical bad element arrived in the
identical batch on every interval and nothing ever converged — logged as "sync
failed", indistinguishable from a hub outage.
Isolation now matches the Akka sibling's split exactly: WireToDomain stays a
strict per-item parser and loses its two batch entry points (their absence is
pinned by a reflection test), while GrpcSecretsHubClient converts each item
inside its own try — the same shape as SecretReplicationActor.Materialize.
A dropped item costs one warning naming the secret and a reason category, and
every well-formed sibling still converges. Only parse-shaped exceptions are
caught (ArgumentException, FormatException); a defect still propagates. A row
whose own name is unparseable is logged under a marker rather than echoing the
untrusted string, and no ciphertext, DEK or credential reaches a log line.
GrpcSecretsHubClient had zero coverage — the existing fake implements the
ISecretsHubReader seam directly, bypassing the header, the deadline and the
conversion, which are the only things that class does. It is now driven for
real through a fake CallInvoker handed to the generated client.
The stale doc comment claiming the reconciler isolated the failure is replaced
by one describing the mechanism that actually exists, and Endpoint gains a note
that a plain-http hub sends the bearer token cleartext (doc only, no validation
change).
Tests 52 -> 67 in the package suite; solution green, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Five review items on the Task 2 server half.
1. GetSecrets now enforces a server-side cap on request.Names, refusing an
over-cap batch with ResourceExhausted BEFORE any store read runs. The
well-behaved follower chunks at 500, but client discipline is not a server
defence — the family precedent (HistorianGateway's max_values) is that the
server declines regardless. The new SecretsGrpcHubServerOptions.
MaxNamesPerRequest defaults to 1000, comfortably above that chunk so a
compliant sweep never meets it, and Validate() refuses a non-positive value
at startup rather than letting it read at runtime as a broken follower. The
refusal states the cap and the received count — neither is secret, and an
operator cannot resize a batch without them — and never a name. The cap sits
behind the interceptor, so it is defence in depth against a misbehaving
token holder, not an anonymous-DoS control.
2. AddZbSecretsGrpcHub now refuses a second registration. The two calls do not
compose: options binding MERGES (a key absent from the second section leaves
the first section's value standing) while the SectionPath stamp is
last-one-wins, so a hub registered twice could serve a token bound from one
section while every failure message named the other. Detected via a private
marker registration. The extension also gains real DI coverage — the intended
single-call path asserted for token/SectionPath coherence through a built
provider, a custom section path stamped end to end, a missing token throwing
from the registration call itself, and the duplicate call throwing.
3. Client-streaming and duplex-streaming denial tests, so all four gated handler
shapes are pinned rather than only unary and server-streaming, plus a
valid-token test that reaches all three streaming continuations.
4. A cancellation test: GetSecrets hands the ServerCallContext token through to
the store, pinned against a fake that records it.
5. The proto's service comment stated the ISecretReplicationHub mapping in prose
that no longer matched the RPC name; it now says GetManifestAsync ->
GetManifest and GetManyAsync -> GetSecrets explicitly. Comment-only — no
message or rpc definition is touched. The GetSecrets rpc comment also notes
the new RESOURCE_EXHAUSTED cap, which a client author has to know about.
Grpc server suite 50 -> 73 tests; full solution suite green, 0 new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Client half of the central-hosted secrets hub: a site node follows central and
never pushes.
- `ISecretsHubReader` + `GrpcSecretsHubClient` — the read half of the hub as a
follower sees it (manifest + bulk fetch, nothing else). Bearer credential and a
per-call deadline are attached per call; responses convert through
`WireToDomain`. Faults are deliberately not caught here, so an unreachable hub
can never be mistaken for an empty one.
- `WireToDomain` — strict, per-row proto -> domain parse mirroring the Akka
replicator's `SecretRowDto`: a name the domain would reject, a content type
naming no enum member, or a non-round-trip timestamp is refused at the boundary.
Explicit proto3 presence keeps null distinguishable from empty.
- `GrpcSecretSyncService` — `SqlServerSecretSyncService` with the push half
removed. One `ReconcileAsync` per tick (pull-only by the reconciler's own
contract, and batched, so a cold-start pull set cannot exceed gRPC's 4 MB cap),
never throws out of `ExecuteAsync`, warns once and retries on the next interval.
The local store is left completely undecorated — the sweep is its only hub-ward
writer.
- `SecretsGrpcHubClientOptions` — fail-closed `Validate()` naming the exact
configuration key, including the section a caller actually bound. The bearer
token is never echoed by any message.
- `AddZbSecretsGrpcHubClient` — binds, eagerly validates, and registers the
channel under a service KEY so it cannot lose a `TryAdd` race to an
application's own `GrpcChannel` and dial the wrong endpoint with the hub
credential attached.
Tests 52/52 (red-first), including a source-scan guard pinning that no client file
names `ComputePushSet`, `ReadLocalAsync`, or `ApplyReplicatedAsync`, and a
reflection pin that the reader seam exposes exactly two read operations.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The server half of the central-hosted pull-only secrets hub. SecretsHubService
adapts the node's own ISecretStore onto secrets_hub.v1 using exactly two read
members (GetManifestAsync, GetAsync) — the proto exposes no write RPC, so a
follower has no path to put a row into central whatever its credentials allow.
GetSecrets omits a name the hub does not hold and skips one that will not parse
as a SecretName; neither is an error, because both describe a follower acting on
a manifest that was true a moment ago. Failing the call would stall the sweep and
let one bad name deny the follower every other row in the batch. Names are
re-validated server-side (the wire type is a plain string) and de-duplicated on
the normalized value.
DomainToWire mirrors the Akka replicator's SecretRowDto discipline exactly:
round-trip "O" timestamps so a DateTimeOffset's offset and ticks survive the
last-writer-wins equality compare, the content type as its enum MEMBER NAME so an
unknown value is rejected rather than decoded to the zero member, and nullable
fields left absent rather than written as "" so null and empty stay distinct.
SecretsHubAuthInterceptor fails closed: with no token configured every call is
denied and one warning names the exact configuration key — never a token value,
presented or configured, at any level. Denial is uniform (one Status, one detail,
no "why") so a caller gets no oracle. Comparison is constant-time over UTF-8
bytes. All four handler shapes are gated, not just unary, so a future RPC cannot
arrive unauthenticated by omission; more than one Authorization header is refused
as a smuggling shape. The type keeps EXACTLY ONE public constructor — Grpc's
activator silently stops invoking an interceptor it cannot construct
unambiguously, and the failure mode is an auth gate that quietly does not run — so
a test pins the count.
Tests: 50, red-first (compile-red, then a fail-closed mutation verified 5 of them
bite). Full Secrets suite 328 passed / 0 failed / 15 live-gated skips, 0 new
warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the third replicator sibling next to Replicator.SqlServer and
Replicator.AkkaDotNet: the central-hosted gRPC secrets hub. This commit is
scaffold only -- project, wire contract, solution wiring -- no service or
client implementation.
The proto mirrors exactly the read half of ISecretReplicationHub
(GetManifestAsync, GetManyAsync). ApplyReplicatedAsync is deliberately NOT
mirrored: in this topology every update originates at central and a site node
is a strict follower, so the narrow contract IS the enforcement -- with no
write RPC on the wire, a mis-wired or compromised site node has no way to push
a row into the hub regardless of what its authorization allows.
Messages are hand-written mirrors of StoredSecret and SecretManifestEntry
rather than serializations of the domain types, following the Akka
replicator's SecretRowDto precedent: the trust boundary stays structural, and
a field added to StoredSecret later cannot start crossing the network merely
by existing. Timestamps are ISO-8601 "O" strings (not google.protobuf.
Timestamp, which cannot round-trip a DateTimeOffset's offset) so a relayed row
is byte-identical to the one central holds; nullable .NET fields use proto3
explicit presence so null and empty string stay distinguishable.
Grpc 2.76.0 / Google.Protobuf 3.34.1 match what ScadaBridge, OtOpcUa and the
sibling ZB.MOM.WW.LocalDb libraries already resolve, so a consumer taking both
gets one Grpc rather than a downgrade warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).
The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.
- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
preserved) + the two surviving false claims struck in place (behavior-spec row and the
Mechanism section's 'races are benign'); points at ScadaBridge
Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
#520 stage 6 (master 2026-08-01, live-gated). With Security:Acl:Enabled ON,
every data-plane operation needs a matching NodeAcl grant, not just writes /
alarm acks / HistoryRead. The stage-5 entry told ScadaBridge that a read-only
consumer was 'unaffected either way' — that now holds only while the flag is
off, which is the default.
Also records the two stage-6 behaviours a consumer would otherwise discover the
hard way: a deep Browse grant makes its ancestor folders browsable (so it
exposes their names), and revoking a grant stops a live subscription — one
BadUserAccessDenied then silence, resuming by itself if the grant returns, with
no reconnect needed.
Default is off, so a read-only address-space consumer is unaffected. With it on,
writes / alarm acks / HistoryRead need a matching grant for the principal the
session authenticates as — anonymous unless ScadaBridge is issued a key.
Reads, browses and subscriptions stay ungated in both modes.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Gitea #520 (OtOpcUa master, merge fbe7403c) replaced LdapOpcUaUserAuthenticator
with an API-key authenticator on the OPC UA data plane. The index still told a
consumer that enforcement was "LDAP roles plus a write gate", which is now the
wrong instruction for anyone wiring a client.
Records what a consumer actually needs: the UserName/`apikey`/`otua_...` token
shape, that LDAP survives for AdminUI login only, that anonymous still defaults
to admitted (so ScadaBridge is not broken today), and that revocation is
asymmetric because keys reach driver-only nodes inside the deployment artifact.
The "ACLs authored but not enforced" warning stands -- stages 3, 5 and 6 remove
it.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
All three live-gated and merged to OtOpcUa master. The index still said write-through was
deferred and every MQTT node read-only, which #508 changed on 2026-07-31; #511 and #515
landed in merge 857a10b4. Two of the three are consumer-visible to ScadaBridge: MQTT nodes
now materialize Operate rather than ViewOnly where a write destination is declared, and a
Sparkplug UDT surfaces as ordinary scalar nodes instead of being skipped.
Both were recorded as open keep-or-delete decisions; both closed on OtOpcUa master abda6f9e.
Neither changes the OPC UA wire contract, so neither is consumer-facing to ScadaBridge — but
the index asserted them as current state, so it would have drifted.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The entry claimed the periodic tag-set drift check had shipped. It was taken back
out two days later and is not in master. Corrected in place rather than deleted,
so the record shows both what landed and what became of it.
Carries forward the consequence a reader needs: AbCip and FOCAS browse a live
backend but implement no IRediscoverable, so they now have no tag-change signal
at all. Galaxy/TwinCAT/MTConnect/Mqtt raise natively and are unaffected. Still
not consumer-facing to ScadaBridge -- no OPC UA wire-contract change either way,
and the IRediscoverable operator prompt is untouched.
Also records the rest of that merge: 13 stale skipped tests retired, a reflection
guard over the deploy-time TagConfig gate, three silently-failing OpcUaClient
integration tests fixed, and the two missing driver guides written.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The index recorded this as unmerged on a feature branch; it landed on master
2026-07-28. Also records the two follow-ups the first pass had deliberately
left unbuilt and which have since shipped: Sql/FOCAS in-place config re-parse
(made atomic, so a config change cannot half-apply a new tag set against an
old connection or backend), and tag-set drift detection for the four
genuinely-browsable drivers.
Neither is consumer-facing to ScadaBridge — no OPC UA wire-contract change, and
the drift signal is advisory rather than an address-space rebuild. The
previously-noted operationally-visible item stands: a DriverConfig edit now
costs a driver reconnect.
Corrects a fact this index recorded two days ago as consumer-relevant to
ScadaBridge. "IRediscoverable is inert fleet-wide" no longer holds: it is
consumed as an operator re-browse prompt on /hosts, carried on
DriverHealthChanged. The consumer-facing conclusion is unchanged and worth
keeping explicit — the served address space still only changes on a config
redeploy, because the prompt is advisory by design.
Gitea #507 (discovered-node injection) is closed as superseded and its code
deleted; it read two structures that are structurally empty in v3, so it could
never have injected anything. #521 tracks the still-unconsumed
IHostConnectivityProbe.
Adds the remediation status for the deferment register (branch
feat/deferment-remediation, tip 88ce8df0, unmerged). One item there is
operationally visible beyond OtOpcUa: #516 makes a changed DriverConfig
stop+respawn the driver rather than an in-place delta, so a config edit now
costs a driver reconnect — previously five drivers ignored the edit entirely
while the deployment still sealed green.
Corrects four stale claims about OtOpcUa and adds two consumer-relevant facts,
all source-verified against the OtOpcUa repo (not its docs):
* "Phases 0a..6 done, 7 not started" -> PROGRAM COMPLETE. Phase 7 merged
c50ebcf7 (6/6 failover drills) + bootstrap guard d1dac87f, both on master
and pushed 2026-07-24.
* The auto-down 1-vs-1 crash-the-oldest live gate is no longer "still open" /
"deferred to Phase 7" — Phase 7 drill D4 closed it; every survivor stayed
Up without self-downing.
* Phase 6's live gate was NOT "deferred by owner choice" — it ran and caught
three fix-forward defects, including a boot-crash StackOverflow on every
node (IClusterRoleInfo resolved inside the AddAkka configurator while the
ActorSystem was still building).
* Records the MAIN-only manual-failover constraint and the default-OFF
bootstrap guard's founder-election shape.
Adds OtOpcUa/deferment.md (e08855fb) and its two ScadaBridge-relevant entries:
* Node ACLs are authored, shipped in every artifact, and NOT ENFORCED — the
evaluator subsystem has zero production consumers and OtOpcUaNodeManager
never references it. Reads are ungated; authored deny rules have no effect.
Do not assume OtOpcUa filters an address space per-user.
* IRediscoverable is inert fleet-wide (#518/#507) — nine drivers raise it,
nothing consumes it, so a Galaxy redeploy / PLC re-download / Agent restart
does not change the served address space without a config redeploy.
Keeps the umbrella index in step with the OtOpcUa driver set, per the
propagation rule in that repo's CLAUDE.md. Read-only Agent driver:
hand-rolled HTTP/XML client (probe/current + multipart /sample pump),
TrakHound dependency evaluated and dropped, browse free via the Wave-0
universal discovery browser, write-back deferred. Live-gated against a
real mtconnect/agent 2.7.0.12 fixture.
Sister projects are told to update their entry here whenever an indexed fact
changes, but that edit lands in this repo's own checkout — so a naive commit
writes it onto whatever branch scadaproj happens to be on. Observed 2026-07-27:
the OtOpcUa Sql driver's entry sat unpushed on feat/overview-dashboard for three
days, and the Mqtt entry was about to join it.
Records the procedure (stash, main, push, restore) and says to cherry-pick
stranded index commits rather than merge the branch they are stuck on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The OtOpcUa entry's driver set changed, so the index has to change with it
(OtOpcUa CLAUDE.md's cross-repo rule).
A read-only `Mqtt` broker-subscribe driver landed with two ingest shapes over
one MQTTnet-5 client: Plain binds a tag to a concrete topic, SparkplugB decodes
vendored Eclipse-Tahu protobuf under a single spBv1.0/{groupId}/# subscription
and binds by the group/edgeNode/device?/metric tuple (birth/alias/seq-gap/
rebirth state machine, death→STALE, a birth-driven browse picker with a scoped
Request-rebirth NCMD). Write-through is deferred — every MQTT node is read-only.
Live-gated on docker-dev against a Mosquitto TLS+auth broker plus a
project-owned C# Sparkplug edge-node simulator on 10.100.0.35:8883.
Recorded as **on branch feat/mqtt-sparkplug-driver, not yet merged to master**,
so the index does not overstate what is on OtOpcUa's default branch — unlike the
Sql DB-poll driver next to it, which is merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
OtOpcUa merged a read-only Sql Equipment-kind driver to master 2026-07-24
(polls SQL Server → OPC UA variables; connectionStringRef env resolution,
schema-walk address picker, live /run gate passed on docker-dev). Keep the
umbrella index's OtOpcUa driver enumeration in sync per the propagation rule.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The OtOpcUa entry's driver set changed, so the index has to change with it
(OtOpcUa CLAUDE.md's cross-repo rule).
A read-only `Mqtt` broker-subscribe driver landed with two ingest shapes over
one MQTTnet-5 client: Plain binds a tag to a concrete topic, SparkplugB decodes
vendored Eclipse-Tahu protobuf under a single spBv1.0/{groupId}/# subscription
and binds by the group/edgeNode/device?/metric tuple (birth/alias/seq-gap/
rebirth state machine, death→STALE, a birth-driven browse picker with a scoped
Request-rebirth NCMD). Write-through is deferred — every MQTT node is read-only.
Live-gated on docker-dev against a Mosquitto TLS+auth broker plus a
project-owned C# Sparkplug edge-node simulator on 10.100.0.35:8883.
Recorded as **on branch feat/mqtt-sparkplug-driver, not yet merged to master**,
so the index does not overstate what is on OtOpcUa's default branch — unlike the
Sql DB-poll driver next to it, which is merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
OtOpcUa merged a read-only Sql Equipment-kind driver to master 2026-07-24
(polls SQL Server → OPC UA variables; connectionStringRef env resolution,
schema-walk address picker, live /run gate passed on docker-dev). Keep the
umbrella index's OtOpcUa driver enumeration in sync per the propagation rule.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The dashboard derived every leader chip from AkkaClusterHealthCheck's `leader`
data — ClusterState.Leader, the LOWEST-ADDRESSED Up member. That is not the node
in charge. The node holding the singletons, gated for writes, and answering 200
on /health/active is the OLDEST Up member. The two agree only until some node
restarts, after which the restarted node rejoins youngest but keeps its address.
Observed live on the rebuilt OtOpcUa rig during the Health 0.3.0 work: leader
central-1, active node central-2. The dashboard would have put the Active chip on
central-1 while /health/active said central-2 — the chip contradicting the tier
it exists to visualise. The Phase 4 acceptance run missed it because the rig had
just been formed, the one state in which the two orderings agree.
Health 0.3.0's active tier now publishes `activeNode` on every member, including
standbys, so the fix is to prefer it and keep `leader` as the fallback for an
instance on an older Health or whose active tier was not probed.
The instance card now shows `active` and `leader` side by side rather than
swapping one for the other: they are legitimately different nodes, and showing
both turns a silent contradiction into something readable at 3am.
Tests: 177 (+3) — precedence when the two disagree, a standby naming the active
node rather than itself, and the fallback still working with no activeNode.
Rewrites the active-node sections of the spec and shared contract, which
described the leader-based design that both consumers rejected.
- SPEC §2.3 replaced: the rule, the ordered RolePreference, why a node holding
none of the preferred roles is Unhealthy rather than Healthy, and the data
payload. §4 (IActiveNodeGate) and the tier table follow.
- SPEC §6 marked historical: it still tells adopters to use
ActiveNodeHealthCheck(role: "admin") and AkkaActiveNodeGate, which is the
advice that produced the defect. Kept for provenance, annotated so nobody
follows it again.
- GAPS: gaps L1/L2 recorded as closed the wrong way, reopened, then closed at
0.3.0 — including the part worth remembering, that both apps adopted the
shared check as specified, hit it live, and reverted to private copies, so the
surface reached zero consumers by being avoided rather than unused.
- Index rows updated with the consumer-visible changes and both app branches.
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.
Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.
Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.
New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.
BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
That "not applicable => Healthy" made the tier answer 200 on every node and
silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
overridable via NoActiveRoleStatus. This case is reachable, not defensive —
OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
correctly a different member during the overlap.
Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.
Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
The shared spec states the active tier "Fails (503) on a standby or
role-member-but-not-leader node", and the same spec maps Degraded to HTTP 200.
ActiveNodeDecision.Evaluate returned Degraded for exactly that case, so the tier
answered 200 on every node and could not distinguish an active node from a
standby over HTTP at all - only the response body differed.
The consequence was live, not theoretical: OtOpcUa's Traefik routes the admin UI
by this probe (docker-dev/traefik-dynamic.yml, scripts/install/traefik-dynamic.yml),
so both central nodes always passed and the leader-pinning the design called for
has silently never worked. Found while running the overview dashboard's live
acceptance, which reads the same tier and showed all six rig nodes as Active.
Filed as lmxopcua#494.
Role-member-but-not-leader now returns Unhealthy. The other two branches are
unchanged: a node that lacks the role stays Healthy (the probe is irrelevant to
it), and the startup-safety path still returns Degraded when the ActorSystem or
cluster is not yet available.
Blast radius is one consumer. OtOpcUa is the only user of the role-filtered mode;
ScadaBridge deliberately uses its own OldestNodeActiveHealthCheck (leadership is
address-ordered and diverges from the singleton host), and MxGateway and
HistorianGateway have no Akka cluster.
Version 0.2.1, three packages published to the Gitea feed and restore-verified.
70 tests pass.
Found by the live rig, and invisible to every unit test that existed.
LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.
Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.
Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.
After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.
174 tests, 0 warnings.
Found during the live acceptance run. The host calls UseStaticWebAssets only when
the environment is literally "Development", so running the build output under any
other name - checking a staging registry locally, say - leaves the Theme RCL's
assets under _content/ unresolvable. The failure is silent and unhelpful: the page
returns 200 with its markup fully present and renders as a blank white screen,
because every stylesheet 404s. Calling it unconditionally is a no-op once
published (the manifest is gone and the assets are real files on disk), so the
only thing it changes is that failure. The container was never affected.
Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass,
2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy.
Notable live evidence:
- Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the
split-brain chip appeared while the pair was self-formed then cleared once it
genuinely joined.
- Kill to Unreachable took two failing sweeps (the damping working as designed);
recovery landed on the first good poll.
- Five consecutive page loads in 0.00-0.02s all showed the same "last sweep"
timestamp, which is cache-first stated as a measurement.
- Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller
mid-sweep and the cards aged into STALE on their own - staleness is computed at
render, so a stuck poller needs no explicit stall detection.
The run also surfaced a pre-existing rig defect in both products: self-first Akka
seed lists mean a pair booted cold self-forms two 1-member clusters that never
join. After a --force-recreate every ScadaBridge node named ITSELF leader and every
node answered /health/active 200; restarting ONE node of each pair converged them
to one Leader plus one Active plus one Standby. That is the user's own open backlog
item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the
dashboard made it visible fleet-wide in one screen.
170 tests, 0 warnings.
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
Tasks 3.4-3.6 of the overview-dashboard plan.
Polling (3.4/3.5):
- OverviewPollerService: one timer at the fastest configured cadence, each
target polled when its own interval is due, fan-out per sweep, no retries
(the next tick is the retry, so damping stays the only place that decides
whether a failure is worth showing).
- OverviewSnapshotStore: atomic swap of a complete immutable graph plus a
change event. Pages read it and never probe — the cache-first requirement.
- LeaderResolver: per-group leader from akka-cluster data, with the
split-brain flag scoped to members that are actually answering, so an
ordinary failover is not misread as a disagreement.
UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup;
instance cards carry the status stripe (dashed for Unreachable), the check
detail list and the raw ready/active signal line.
Two defects found by verification rather than by review:
- The active tier answers with a status code and an EMPTY body, but the client
demanded parseable JSON on every probe — so a healthy pair rendered as
"role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier,
body is the payload) and ProbeStatusAsync (active tier, code is the answer).
Caught by a live smoke run against two fake health endpoints; the poller
tests now serve an empty body so they hold the fix.
- SweepAsync published even when cancellation was already requested, leaving a
half-probed registry as the store's final state on shutdown. It now checks
the token itself rather than relying on the transport to throw.
Also: app.UseAntiforgery() is required despite the anonymous-by-design
pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and
the endpoint middleware hard-fails without it (every page was 500). Chosen
over DisableAntiforgery() so a future form is protected by default.
147 tests, 0 warnings. Live-verified end to end against fake endpoints:
Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
Phase 3 tasks 3.1-3.3 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md.
Scaffold (3.1): a plain directory in scadaproj (NOT a nested git repo), slnx over
src/ + tests/, HistorianGateway-pattern Directory.Build.props (warnings-as-errors)
and nuget.config (nuget.org * + the Gitea feed scoped to ZB.MOM.WW.*). Inline
package pins, no CPM — the app convention here. NO Auth/Audit/Secrets packages:
the dashboard is anonymous and read-only by requirement.
Registry (3.2): OverviewOptions/ApplicationEntry/InstanceEntry bound from the
"Overview" section, plus EffectiveTimings resolving instance > application >
global overrides in ONE place so no caller re-implements the precedence.
OverviewOptionsValidator (OptionsValidatorBase + AddValidatedOptions) rejects an
empty registry, applications with no instances, duplicate names, non-absolute or
non-http(s) URLs, and non-positive intervals — and checks stale > poll on the
EFFECTIVE values, since an override at either level can invert that on one
instance while the globals look fine. A ConfigPreflight presence check runs before
the host is built so a missing registry fails with the key name, not with
"Applications must have at least 1 entry".
Health client (3.3): ZbHealthReport DTOs for the canonical ZbHealthWriter body,
with per-entry `data` kept as JsonElement — it is an open contract, and binding it
to concrete types would break the moment a check adds a field. 200 AND 503 are
both parseable answers (503 carries the body naming the failing check); only a
transport failure or a non-canonical body is a failed probe, which is why
Unreachable stays distinct from Down. Host shutdown propagates as cancellation
rather than being recorded as every instance timing out.
Status model: Up/Degraded/Down/Unreachable + Active/Standby/Unknown/NotApplicable,
with two-strike flap damping — a failing observation only replaces a good state
after 2 consecutive failures, while recovery is immediate (damping suppresses
false alarms; symmetric damping would just make them linger).
53 tests green, 0 warnings in Debug and Release. Tests for the validator, the
client golden payloads (with AND without `data`, so a partly-bumped fleet renders)
and the derivation table are front-loaded here from task 3.8 so this batch is
verified rather than pending.
Also pins AngleSharp 1.5.2 test-side: bunit 1.40.0 pulls 1.2.0, which trips NU1902
and therefore the warnings-as-errors gate (same fix HistorianGateway made in
6bc005d).
Phase 0 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md: give the
canonical health JSON a structured channel so the family overview dashboard can
read each Akka cluster's current leader.
- ZbHealthWriter: optional `"data": {...}` per entry, sourced from
HealthReportEntry.Data, emitted only when non-empty. Per-property JsonIgnore
(NOT a global DefaultIgnoreCondition) so `"description": null` still renders —
payloads from data-less checks stay byte-identical to 0.1.0.
- AkkaClusterHealthCheck: BuildClusterData publishes this node's own view —
leader (omitted while unknown), selfAddress, selfRoles (sorted), memberCount,
unreachableCount — on every result path. The startup-safety paths (no
ActorSystem / cluster inaccessible) stay description-only.
- Tests: writer data emit/omit (raw-JSON assert on the omit case), and a real
single-node self-joined cluster via Akka.TestKit.Xunit2 for the data values.
70 tests green (25/39/6).
- Version 0.1.0 -> 0.2.0; 3 packages published to the Gitea feed and
restore-verified from a scratch consumer, which serves data.leader live.
Per-cluster mesh Phase 6 merged to OtOpcUa master 2839deb5 (pushed 2026-07-24).
Records the split into one 2-node mesh per application Cluster, the consumer-visible
per-Cluster ServiceLevel/Primary scoping (two+ Primaries fleet-wide by design, no
wire-contract change), the SplitTopologyTransportValidator, pair-local secrets, and
the deferred live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 merged + pushed (tip 3a590a0c): driver-only nodes run with no ConfigDb
connection string. Consumer-visible ServiceLevel change — a DB-less site node
now holds 240/250 with central SQL down (survive-alone), where a DB-backed node
would drop to 0/100. Phases 0a/0b/1/2/3/4 done; 5-7 not started.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Update the OtOpcUa entry: the per-cluster mesh election-scoping fix now has Phases
0a/0b/1/2/3 merged + pushed (tip d01b0695). Phase 2 = ClusterClient command transport
(MeshTransport:Mode dark switch). Phase 3 = config fetch-and-cache — a FetchAndCache
driver node boots + serves last-known-good with central SQL down (extends the #485
availability guarantee consumers like ScadaBridge rely on); a failed fetch is 'no answer',
never an empty config. Live gate caught + fixed a Status==Sealed serve-gate deadlock.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Options investigation (Aspire standalone/HealthChecks.UI/Gatus/Homepage/
Grafana vs custom) → decision: custom ZB.MOM.WW.Overview Blazor app in the
family look. Design: anonymous read-only single pane, appsettings registry
of all four apps' instances, cache-first poller on a configurable timer,
configurable staleness timeout, Active/Standby from /health/active, and
per-cluster Akka leader chips (needs Health 0.2.0 optional per-entry data).
Impl plan: phased + code-verified — Phase 0 Health 0.2.0 (writer data field
+ AkkaClusterHealthCheck cluster view), Phase 1 ScadaBridge site-node
MapZbHealth PR (:8084; role-scoped active check), Phase 2 OtOpcUa bump,
Phase 3 the app (:5320), Phase 4 live rig acceptance. Mockup is the visual
reference: real Theme tokens + embedded IBM Plex, all card states incl.
stale + split-brain. Ready to execute in a fresh session.
Corrects the same-day report: PR searches for 'quic' missed work named
'artery'. Artery is landing rapidly on upstream dev as Artery-TCP (openspec
#8313; framing/handshake G1-G2; parallel lanes #8356/#8357; large-message
streamId 3 #8352; MNTR/CI for both transports), with Serialization.V2 in
parallel. TCP-first resolves §7's forward note: pipeline benefits arrive
without libmsquic, dropping the §6 platform constraints from the 1.6
adoption path. No previews or public benchmarks yet.
Two research reports (2026-07-22):
- akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take,
the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native
alternatives (managed sockets ranked first), and measured throughput
evidence (PR #4594: transport swap is throughput-neutral).
- akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the
central-pair + site-pairs topology, the stalled Akka.NET port (#3261 /
PR #4111), per-system fit vs the OtOpcUa mesh program, and the
before-Phase-2 decision point.
Recommendation in both: don't build; track upstream.
Both are consumer-visible, so they belong in the index rather than only in the
repo. The downing strategy moved to AutoDowning because a two-node OtOpcUa pair
on keep-oldest could not survive a crash of the oldest node — the same defect and
the same fix as ScadaBridge cf3bd52f, which is where the analysis came from. And
the redundancy Primary is now the oldest Up driver member rather than the role
leader, which changes which node advertises ServiceLevel 250 after a restart —
relevant to anything selecting a server by ServiceLevel.
Also notes that the four NU1903 advisories which broke ScadaBridge's fresh
restores were breaking OtOpcUa's too, and that the election remains scoped per
Akka cluster until the per-cluster mesh work lands.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The #485/#486 fixes changed nothing structural about OtOpcUa — same drivers,
same dual-namespace scheme, same Galaxy flow — so the row is mostly untouched.
What DID change is a fact any OPC UA consumer depends on, and ScadaBridge is
mid-cutover onto exactly this address space: a transient ConfigDb error used to
empty the served address space (a PureRemove of every node, observed live and it
stayed empty), stop every driver and clear every subscription, while still
reporting the deploy Applied. A consumer would have seen all bindings go bad
with nothing upstream admitting a problem. Recorded on the OtOpcUa row.
Also fixes real drift found while checking: the OtOpcUa LocalDb adoption design
was sitting UNTRACKED here and unreferenced by the index, even though the
LocalDb row claims the adoption designs live in this repo. Now committed,
referenced alongside the ScadaBridge one, and its stale header corrected —
it still said Phase 1 was "not yet executed" against lib 0.1.1, when Phase 1
shipped and merged against 0.1.3. Phase 2 designed but not executed, on both
the doc and the row.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two defects about a node whose LocalDb file is lost, the second only reachable
once the first was fixed:
- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
detection read an empty oplog as 'no gap possible'. The steady state of a
healthy pair was the one state from which a rebuilt node could never be
back-filled.
- 0.1.3 — with back-fill working, the rebuilt node's OWN writes were silently
dropped: last_applied_remote_seq is a watermark in the peer's seq space and a
rebuilt peer numbers from 1, so the healthy node's stale watermark made the
sending side skip its whole oplog.
Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig and
reproduced offline as RED tests first. 149/149 pass.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.
last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:
- The healthy node kept advertising the OLD peer's watermark, and the sending
side seeds its pump from exactly that number, so the rebuilt node skipped its
entire oplog. The watermark (and the observed peer clock) is now reset when
the peer's node id changes. last_acked_seq is deliberately NOT reset: it
describes our own oplog's pruned horizon and is what tells a rebuilt peer it
needs a snapshot — clearing it would turn 0.1.2's back-fill back off.
- A peer claiming to have applied more of our stream than we have ever produced
can only be remembering a previous incarnation of us. That claim is now
rejected in favour of starting from the beginning; LWW makes the re-send
harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.
Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.
Version 0.1.3.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The lib row still claimed ScadaBridge was the only consumer (verified
2026-07-20); OtOpcUa's Phase 1 adoption merged 2026-07-21. Records 0.1.2 and
the wiped-peer snapshot fix it carries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the
steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync
measured the peer's gap against the oldest surviving oplog row and treated an
empty oplog as "no gap possible", which made that steady state the one state
from which a wiped peer could never be healed: it reconnected with an empty
database and a zero watermark, was told it needed no snapshot, and stayed
empty indefinitely.
With an empty oplog the oldest seq we could still stream is one past
last_acked_seq — ack-pruning is the only thing that empties the oplog without
also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq.
Measuring the gap against that heals the wiped peer and leaves every other
case identical: a converged peer's watermark equals last_acked_seq (no
snapshot), and a pair that has never written anything has both at zero.
Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as
a documented limitation. Covered by a test that reproduces the live-gate
scenario through the real session stack: RED before this fix (the wiped side
never converges, 15 s timeout), green after. 148/148 pass.
Version 0.1.2.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phases 1 and 2 landed on ScadaBridge main together as PR #23
(merge 28ca04d7) on 2026-07-20 and were pushed to origin. The
component row still described both as unmerged branches.
Verified before editing: main..feat/localdb-phase1 and
main..feat/localdb-phase2 are both empty, main == origin/main,
and SiteReplicationActor/ReplicationService have no hits left in
main's tree.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The row still said "Adopted by ScadaBridge (Phase 1 only)" and "Phase 2 is NOT
started". Both phases are now complete with a 10-check live gate PASS.
Also records what the row could not have said before: the 0.1.1 parent-
directory boot fix, that ScadaBridge is the library's ONLY consumer (verified —
OtOpcUa, mxgw and HistorianGateway have no LocalDb reference at all), the
deliberate never-registered status of notification_lists/smtp_configurations
and why, and the three operational constraints an operator has to know
(stop/start a pair together, the TombstoneRetention resurrection bound, and
MaxBatchSize batching by row count against a 4 MB gRPC cap).
Replication remains default-OFF and no production deployment has it enabled.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes criterion 5 of ScadaBridge/docs/known-issues/2026-07-20-localdb-disk-io-
error-under-load.md §9: Phase 1 shipped only correctness and convergence tests,
so nothing exercised many pooled per-operation connections writing a REGISTERED
table (capture trigger + zb_hlc_next UDF) concurrently — which is exactly the
shape the ScadaBridge site nodes run under load.
8 writers x 250 inserts, one fresh pooled connection per operation, with
concurrent readers churning the pool and the WAL index throughout. Asserts both
that every write lands and that the oplog captured each exactly once, so a
trigger that dropped or double-counted under contention fails it.
Written during the 2026-07-20 disk-I/O incident investigation and left
uncommitted; the incident itself was root-caused to host-side sqlite3 access on
a bind-mounted WAL database, not a library defect.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
SQLite creates the database FILE on demand but never its parent DIRECTORY, and
SqliteLocalDb opens the file eagerly in its constructor — so a configured path
under a not-yet-existing directory was a hard startup failure ("SQLite Error
14: unable to open database file") rather than a first run that just worked.
Relative defaults like ./data/app.db hit it routinely; a containerized consumer
escapes only when a volume mount happens to create the directory first.
ScadaBridge found this during LocalDb Phase 2 and worked around it app-side
(SiteLocalDbDirectory.Ensure). Every other consumer still had the gap, so the
guarantee belongs here.
The standing objection to a library that creates directories is that it can
mask a mis-typed path by silently starting an empty database. That does not
apply: SQLite ALREADY does exactly that for a mis-typed file NAME. Refusing to
create the directory protects nothing — it only makes the two halves of one
path behave inconsistently and turns the directory half into an opaque error.
A creation failure (permissions, read-only mount) now throws an
InvalidOperationException naming the directory and the cause, instead of
surfacing as SQLite error 14. That is a change of exception type, but only on a
path that already failed hard.
Both behaviours are pinned and verified to fail with the call removed.
147 tests pass, 0 warnings. Published to the Gitea feed at 0.1.1.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Was 'no app adoption yet'. Now records the honest state: ScadaBridge site nodes
adopted Phase 1 on an UNMERGED branch, replication wired but default-OFF and
live-proven on the docker rig's site-a pair only, Phase 2 not started, no other
app adopted, no production deployment replicating.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Fixes scadaproj#2. ConfirmDeleteModal used the bare `modal` class and the
Secrets.Ui RCL shipped no CSS at all, silently depending on the host having
no opinion about that selector. Bootstrap 5 does: `.modal { display: none }`
made the delete modal permanently invisible on every Bootstrap host
(ScadaBridge, OtOpcUa, HistorianGateway — confirmed live on the first two)
while every @onclick handler kept working. Only MxGateway, the lone
Bootstrap-free host, could ever have rendered it.
Fix, entirely inside the RCL so a package bump repairs every host with no
host-side changes:
- class vocabulary renamed to zb-secrets-modal / -backdrop / -card / -title
so no host framework selector can match the elements;
- the component emits its own <style> block alongside the markup. Scoped
.razor.css was deliberately NOT used: three of the four family hosts never
link a scoped-CSS bundle, so isolation CSS would silently fail to load —
the exact defect class being fixed. Theme tokens with fallbacks keep the
card legible even on a host without the kit stylesheet.
- data-testids unchanged, so existing Playwright gates and bUnit tests keep
their selectors.
New bUnit pins: no Bootstrap-reserved class name appears in the rendered
markup, and the component ships a style block that both positions the
overlay and gives it a display mode. Suite 182 pass / 0 fail / 0.2.3.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
ActorContextAfterAwaitTests asserted that a ConfigureAwait(false)
continuation on the shared thread pool always throws NotSupportedException
when reading Self/Context. That property is a timing accident, not an Akka
guarantee: pool threads are exactly where Akka and the TestKit legitimately
install the [ThreadStatic] actor cell during mailbox runs and async
continuations, and Akka 1.5.62 has two non-throwing states besides — a
cleared cell makes ActorBase.Context return null (NullReferenceException on
.Self, not NotSupportedException) and ActorBase.Self return _clearedSelf
without any throw. Under parallel suite load the assertion failed once at
exactly that seam (2026-07-18); ironically the test's own doc comment said
the behaviour "does not reliably throw" and then asserted reliability.
The illegal reads now run on a dedicated new thread (LongRunning), the one
place the no-context state is guaranteed, while the realistic
ConfigureAwait(false) escape from the mailbox is kept. If the reads ever
unexpectedly succeed again, the failure message reports whose context the
thread was carrying. Verified 3 consecutive full-project runs green (38/38)
after 8 instrumented runs hunting the original repro.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
scadaproj#1 holds the defect (the library lives here); lmxopcua#482 is the
consumer-side tracker keeping Secrets:Replication:Enabled=false until it is fixed.
SQL-Server hub topology (ScadaBridge) is live-validated against a real SQL Server
2022 with two nodes: write->resolve via hub, tombstone propagation without
resurrection, bidirectional sweep, partitioned-node local resolution (site
autonomy - the reason hub mode was chosen), and wrong-KEK fail-closed via
SecretDecryptionException.
Akka peer-to-peer topology (OtOpcUa) FAILS the gate: resolving ISecretReplicator
hangs indefinitely in a real clustered process. Reproduced twice on a healthy
2-node cluster; isolated to constructing AkkaSecretReplicator -> ActorRef ->
system.ActorOf, NOT nested DI resolution. The library's own 2-node suite passes,
so the trigger is creation through DI in a hosted process. Root cause not yet
identified.
Impact: OtOpcUa would hang at startup if replication were enabled, since the
startup hook resolves ISecretStore. Harmless today only because the flag defaults
false. Do NOT enable Secrets:Replication:Enabled in OtOpcUa.
The gate did its job - it caught a blocking defect that every offline suite,
including the library's own real-cluster tests, had passed over.
- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
Task 9 remains open and the topology is not 'adopted' until it passes.
AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(). The package's own
TryAddSingleton<ISecretReplicator> therefore found a descriptor already present
and was silently discarded.
Consequence: ISecretReplicator resolved to the no-op sink, so every write
published into nothing; and because SecretReplicationActor is only spawned as a
side effect of constructing AkkaSecretReplicator, no actor was ever created
either. No exception, no log line - a cluster that reports healthy and silently
never converges. The worst available failure mode for a secrets store.
Fix: register ISecretReplicator BEFORE AddZbSecrets, matching what the SQL-Server
package already did. Found during OtOpcUa adoption (Task 6), which is the first
code that ever built a container around this extension.
Root cause of the gap: the SQL-Server package had a DI test asserting its
replicator type (AddZbSecretsSqlServerTests:58) and the correct order; the Akka
package had neither. Every Akka test exercised the actor, serializer and
reconciler in isolation - none built a container, so nothing could see it. This
is the third instance of the same defect class in this library (the inert
ISecretReplicator seam, the unregistered concrete SqliteSecretStore, and now
this), all of which share one cause: unit tests that never construct the DI graph.
Adds AddZbSecretsAkkaReplicationTests (5 tests) asserting registration at the
ServiceCollection level. Verified to discriminate: with the 0.2.0 order restored,
2 of the 5 fail; with the fix, all 5 pass. They assert descriptors rather than
resolving from a provider on purpose - resolving ISecretReplicator eagerly spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
which would make the test hang rather than fail.
Full suite Release-green: 175 passed, 15 skipped, no new warnings.
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.
The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.
Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.
170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes out the security-fix rollout: mxaccessgw, ScadaBridge and HistorianGateway
are all on Auth 0.1.5 and pushed. Exposure was checked per repo rather than assumed --
only mxaccessgw and ScadaBridge actually resolved the vulnerable 2.1.11;
HistorianGateway already resolved 3.50.3, and OtOpcUa takes no ApiKeys/SQLite
dependency at all.
Also records two things worth not rediscovering: HistorianGateway's bump is
unverified by tests, and both it and ScadaBridge have a pre-existing AngleSharp
NU1902 error failing full-solution builds under TreatWarningsAsErrors.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Packing Secrets 0.1.3 surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls
SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries high-severity advisory
GHSA-2m69-gcr7-jv3q. Auth.ApiKeys had the same exposure and was already shipped
at 0.1.4 to three consumers, so both libs are fixed rather than only the one
being published.
Fixed the way ZB.MOM.WW.LocalDb already had it: CentralPackageTransitivePinningEnabled
plus a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help --
even 10.0.10 still resolves 2.1.11 -- so the pin is the actual fix. The pin reaches
consumers: both nuspecs now declare SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct
dependency, verified by restoring the published packages into a scratch project,
which resolves 2.1.12 and scans clean.
Auth goes 0.1.4 -> 0.1.5 with no API change (0.1.4 plus the pin). Suites re-run
after the native-lib swap with identical counts, so no behavioral regression:
Secrets 97 pass/1 skip, Auth 215 pass/1 skip (both skips are Windows-only DPAPI
and opt-in LDAP).
Consumers are NOT yet bumped and remain on vulnerable versions: mxaccessgw 0.1.4,
HistorianGateway 0.1.4, ScadaBridge 0.1.3. Secrets consumers all sit at 0.1.2 and
also lack KEK rotation.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified
from a scratch consumer) and adds the build/push.sh the other shared libs already have.
The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from
reality, so it was re-derived from the feed listing and the actual PackageReferences +
registration calls on each consumer's default branch rather than from prior claims.
Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth),
GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires
AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet
committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said
"three apps" while HistorianGateway is a fourth consumer of seven libs, and all
eight libraries' test counts were stale (re-ran each suite; all green).
Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation
committed, but the feed tops out at 0.1.2, so no app can consume rotation until it
is published.
Health and observability divergence tables are labelled historical, not re-verified —
the libraries are proven wired, but per-app probe coverage vs spec was not re-walked.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
(bodies never re-encrypted; revision/timestamps preserved -> invisible to
cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
updates only the 4 wrap columns + kek_id, compare-and-swap on the current
wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
JSON counts report; README section + operator runbook.
Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).
G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.
Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
Record the OtOpcUa live wonder gate (item 2). A throwaway harness drove OtOpcUa's
real GalaxyDriverBrowser (secret: ApiKeySecretRef resolved via ISecretResolver) against
the real production MxGateway: dummy secret: ref -> MxGatewayAuthenticationException,
real metadata:read key -> CONNECTED + browsed the real Galaxy root (10 nodes). Temp key
minted via the dashboard, then revoked (functionally proven) + deleted; no prod key
touched; harness + scratch stores destroyed. All four apps now G-2..G-6 done + live-proven.
Record the OtOpcUa secrets adoption (G-2/G-4/G-5/G-6, the last app): FF-merged +
pushed to lmxopcua origin/master @872cf7e3. Update the components/secrets/GAPS.md
OtOpcUa bullet (execution record + the three verified plan deviations + the regression
the gate caught) and the CLAUDE.md secrets-component-table row. All four apps
(HistorianGateway, mxaccessgw, ScadaBridge, OtOpcUa) now adopt ZB.MOM.WW.Secrets.
The v3.0 dual-namespace rewrite (PR #472) moved anchors and reshaped the
build, so the plan is refreshed to stay executable: CPM correction (OtOpcUa
is on Central Package Management, not inline versions), OpcUaClient options
in .Contracts + GalaxyDriverBrowser in .Browser, DriverHostActor line
shifts, and appsettings/owner precision. Driver-secret flow itself was
untouched by v3.0, so the two-layer approach holds.
Update the OtOpcUa umbrella entry for v3.0: the single .../ns namespace is
replaced by .../raw (s=<RawPath>) + .../uns (s=<Area>/<Line>/<Equipment>/<Eff>),
single-source fan-out to both NodeIds, Organizes UNS->Raw, writes/HistoryRead via
either NodeId, multi-notifier alarms, EquipmentNodeIds retired. Flags the wire-
contract change as a pending ScadaBridge Data-Connection-Layer cutover.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Add components/secrets/ (SPEC, realized shared-contract, code-verified
current-state for OtOpcUa/mxaccessgw/ScadaBridge, GAPS adoption backlog) and
register the Secrets row in CLAUDE.md + components/README.md. The
ZB.MOM.WW.Secrets lib is built + published 0.1.2 + reference-consumer-proven
(HistorianGateway, live vs the wonder historian); per-app adoption is the
tracked follow-on.
7th shared ZB.MOM.WW.* library: AES-256-GCM envelope-encrypted secret store
(ISecretStore/SQLite), pluggable IMasterKeyProvider, ${secret:name} config
resolution, ISecretResolver runtime cache, Blazor /admin/secrets management UI
on the Theme kit, and a headless 'secret' CLI. Published 0.1.2 to the Gitea feed;
HistorianGateway adopted as reference consumer (live-proven end-to-end against
the real wonder historian). 80 tests pass.
Live HistorianGateway boot caught that the expander scanned a '_secretsComment' appsettings
value documenting the ${secret:...} syntax, tried to resolve the literal example token, and
crashed boot. The family uses _*Comment keys pervasively, so skip any config key whose leaf
segment starts with '_' (the _comment convention). Functional keys never lead with '_'.
Reference-consumer (HistorianGateway) caught that SecretsAuthorization.AdminRole was
'administrator' (lowercase) while the family canonical role is CanonicalRole.Administrator.
ASP.NET role claims compare case-sensitively (Ordinal), so a real admin got 403 on the
secrets UI. Now nameof(CanonicalRole.Administrator).
description: Use when you need the detailed change history, live-gate records, redundancy/auth/ACL behaviour, or ScadaBridge consumer-impact notes for a sister project (OtOpcUa, HistorianGateway, BTDB) — the scadaproj root CLAUDE.md keeps only a short summary row per project; the full narrative lives in this directory.
---
# Sister project history
The root `CLAUDE.md` index keeps one short row per sister project. The full narrative rows that used to live
in that table were moved here **verbatim** on 2026-09-03 (they were ~57k characters loading into every session):
-`otopcua.md` — OtOpcUa: v3 dual-namespace address space, every driver, redundancy/mesh program, data-plane auth + ACL enforcement, deferment register, the authoring-vs-deploy-gate parity arc, and the per-issue consumer-impact record for ScadaBridge.
-`btdb.md` — BTDB (Batch Tracking Database, added 2026-09-03): schema + sproc write contract, Historian/CLR coupling, Galaxy/MESAPI/ScadaBridge edges, the DARS `BatchTracker_DB` fork, and the gap analysis.
Read the file for the project you are working against. When a sister repo tells you to "update its entry in
`scadaproj/CLAUDE.md`", append the detail to the file here and keep the index row to a few lines.
# BTDB (Batch Tracking Database) — index history + analysis
> Added 2026-09-03 as a sister project. **This file is the authoritative place for BTDB detail — append here and keep the root index row short.** Everything below was verified from source on 2026-09-03 unless marked _(inferred)_.
| Project | Location | Stack | Repo | Summary |
|---|---|---|---|---|
| **BTDB** (Batch Tracker) | `~/Desktop/btdb` (case-insensitive FS: also `BTDB`) | SQL Server 2016+ SSDT project + SQL CLR (net472) · manager: **Blazor Web App net10.0** on `feat/manager-blazor` (§8; `master` still carries the EOL net6.0 MVC app) | **`ZimmerBiomet/BTDB`** on the corporate Gitea `wonder-app-vd03.zmr.zimmer.com:3000` (`master` @ `d147c10e`, 2023-07-26). **Not** on `gitea.dohertylan.com`. | The `BT` SQL Server database that is the plant's **batch/cycle/work-order system of record for data collection**: Galaxy machine objects call its `StartMachineBatch`/`StartMachineCycle`/`End*`/`AddMachineBatchWorkOrder`/`AddOtherData*` sprocs; reports pull time-series per batch out of the AVEVA Historian through its CLR sprocs (`OPENQUERY(INSQL, … WideHistory …)` + `Runtime.dbo.*`); MESAPI and ScadaBridge's inbound API resolve `SAPID → Machine.Code` from `dbo.Machine`. `BTDBManager` is a small Windows-service MVC site for machine/group config, batch edits (audited) and cross-environment config import. |
## 1. What is on disk, and where the truth is
-`~/Desktop/btdb` is **not a git checkout** — no `.git`, no `CLAUDE.md`, no `README`. It is a file-for-file snapshot of `ZimmerBiomet/BTDB``master` (the only branch) at its head commit `d147c10e` (2023-07-26, "Update notification system tables to add urgent flag"). Verified by comparing the Gitea tree with the local tree: 170 vs 168 files, the only difference being the remote's `.gitattributes`/`.gitignore`. The repo's `updated_at` of 2024-11-07 is not a commit (the branch head is still 2023-07-26).
-`~/Desktop/DARS/WW_ARCH/BatchTracker_DB/` is a **fork of the `Database` project only** (no Manager), modified 2026-08-13, also not under git (DARS is not a repo). Substantive differences vs the snapshot, all matching the DARS sprint-1 spec `DARS/sprints/sprint1/bt_db_changes.md`:
- new table `MachineBatchTransfer` (source/dest batch chain for mid-run reactor failure; `TransferType` CHECK `'mid_run_failure'`; three indexes) + sproc `AddMachineBatchTransfer` (linear-chain guard);
-`MachineBatchWO` gains `ContainerID NVARCHAR(64) NULL` + `MoveOutTimestamp DATETIME2 NULL` with filtered indexes (Camstar container traceability; mirrors the `SAPID`/`ZTag` idiom);
- new CLR sproc `GetLinkedBatchTSV` (+ `_Wrapper`) that walks the transfer chain and stitches TSVs, and `LinkedBatchAlarm_Wrapper`;
-`StartMachineBatch.sql`, `MachineBatch.sql`, `AuditLogMachineBatch.sql` differ **only** in BOM/CRLF → LF normalization (no logic change).
- **Resolved 2026-09-03 — the divergence is closed locally:** `~/Desktop/btdb` was turned into a real checkout (`git init` + fetch of `ZimmerBiomet/BTDB`, `master` tracking `origin/master`; the snapshot was tree-identical apart from exec bits copied from Windows, which were cleared). The DARS fork's transfer-chain work was migrated onto branch **`feat/machine-batch-transfer`** as one commit: the five new objects, the two `MachineBatchWO` columns, and the `.sqlproj` wiring **including the `Compile` entry for `GetLinkedBatchTSV.cs` that the fork had omitted** (without it the CLR sproc is never built and its wrapper fails at deploy). The BOM/CRLF re-saves were not carried over. The sprint-1 plan (`bt_db_changes.md`, overview, diagrams) and `BatchTracker_Reference.md` now live under `btdb/docs/`. **The branch is local only — not pushed to the corporate Gitea.**`DARS/WW_ARCH/BatchTracker_DB/` is now superseded and should be deleted or replaced by a pointer once the branch is pushed. A second commit on the same branch carried the two columns through the Manager's batch-edit path: `MachineBatchWOType` (appended last — the TVP binds by ordinal and `ToDataTable` emits properties in declaration order, so the view model's property order must keep matching), `AuditLogMachineBatchWO` (`Prev_`/current pairs), the DTOs and view model, `SaveBatchEdit.sql` (MERGE change-detection with `COALESCE` because both columns are NULL-able, OUTPUT and the three audit inserts), and the Kendo work-order grid in `Views/BatchEdit/Edit.cshtml` (masked timestamp editor like the cycle grid). `LookupBatchInfo.sql` is `SELECT *` so it needed nothing. Not touched: the Telerik `BatchEditAuditLogReport.trdp` (binary package; open it in the Telerik designer to add the audit columns) and the batch search grid (`SearchBatches.sql`), which still lists only work-order numbers. Nothing here was built or run: the Manager needs Telerik + Windows, and the SSDT project needs MSBuild; a throwaway compile of the Manager retargeted to .NET 10 on the Mac showed no errors in the touched files.
| `Manager/BTDBManager.csproj` (**`master` only — deleted on `feat/manager-blazor`, see §8**) | ASP.NET Core MVC, **net6.0** (out of support since 2024-11), Dapper 2.0, `System.Data.SqlClient`, `System.DirectoryServices` (Windows-only LDAP), `Microsoft.Extensions.Hosting.WindowsServices`, Kendo UI + Bootstrap, **Telerik Reporting R2 2022** referenced by an absolute `HintPath` into `Program Files (x86)` | Kestrel `http://*:9876`; runs as a Windows service; 13 embedded `.sql` query resources; 3 `.trdp` Telerik audit reports. `Manager/install.bat`/`uninstall.bat` are **leftovers from a different service** (`WWNotificationSystem.exe`) and do not install the Manager. |
## 3. Database model (what the family writes and reads)
**Core hierarchy:**`Machine` (`Code` unique; optional unique `ZTag`, `SAPID`; `TimeZone`; `MultipleBatch`/`MultipleCycle` flags; `OtherData` JSON) → `MachineBatch` (start/end operator + UTC timestamps, `OtherData`) → `MachineCycle` (same shape) and `MachineBatchWO` (`WorkOrderNumber`, `ItemNumber`). `MachineGroup` + `MachineGroupMembership` give roll-ups (DARS models one `Machine` per reactor *side*, one `MachineGroup` per unit). `MachineAlarm` is a per-machine alarm catalog (`Severity` 0–999, `FlaggedForMES`).
**Lifecycle sprocs (the write contract):**
| Sproc | Behaviour |
|---|---|
| `StartMachineBatch(@Code, @OperatorID, @o_MachineBatchID OUT)` | Looks up machine by `Code`; unless `MultipleCycle`/`MultipleBatch`, **auto-closes** any open cycle/batch first; inserts with `GETUTCDATE()`. Throws 50000 if machine unknown. |
| `EndMachineBatch(@MachineBatchID, @EndOperatorID)` | Closes the open cycle too; throws if not found / already closed. |
| `StartMachineCycle(@MachineBatchID, @OperatorID, @o_MachineCycleID OUT)` / `EndMachineCycle` | Cycle must belong to an open batch. |
| `AddMachineBatchWorkOrder(@MachineBatchID, @WorkOrderNumber, @ItemNumber, @o OUT)` | Only on open batches. |
| `AddOtherDataMachineBatch` / `…MachineBatchWorkOrder` / `…MachineCycle (@ID, @DataKey, @DataValue)` | `JSON_MODIFY` into `OtherData` — via **dynamic SQL that concatenates `@DataKey`/`@DataValue`** into the statement (`sp_executesql` of a built string). Injection surface; callers today are trusted Galaxy scripts. |
All timestamps are `GETUTCDATE()`; `Machine.TimeZone` is used only at retrieval time (`wwTimeZone=` in report wrappers). This matches the family's all-UTC rule.
**Historian coupling (the read contract):** the CLR code resolves a machine-relative tag (`<Code>.<local>`; `*.<attr>` shorthand; `|literal|` static columns) to the Historian's global tagname via `Runtime.dbo.TagExtendedPropertyInfo` (`HierarchicalName`), then runs `SELECT … FROM OPENQUERY(INSQL, 'SELECT DateTime, [tag]… FROM WideHistory WHERE … wwTimeZone="UTC" AND <queryParams>')` (`wwRetrievalMode`, `wwResolution`, `wwTimeZone` are passed through **verbatim** from `@queryParams`). Discrete checks read `Runtime.dbo.DiscreteHistory`; alarm wrappers read `Runtime.dbo.Events` (`IsAlarm=1`, `Alarm_OriginationTime`, `Alarm_DurationMs`, `Alarm_UnAckDurationMs`). **Consequence: the `BT` database must live on (or three-part-name-reach) the Historian's SQL instance with the `INSQL` linked server** — it is co-tenant with `Runtime`, which is why it sits on the `wonder-sql-*` boxes.
**Report wrappers** are per machine family and are the consumers of the CLR layer: Crest (`CrestWide`/`CrestMinMax`/`CrestAlarm`/`CrestCleaner*`), Elma (`ElmaData`, `Elma_Bulk_Query_Nitric`), `ExtruderWide`, `PultrusionWide`, Haas (`HaasStatusReport`, `HaasAlarm`, `QV_Haas*` = QlikView), Solar (`SolarAlarms`/`SolarEvents`), DryHeat (`DryHeatCycleReport`/`Search`/`BulkReport`), PoncePolyPress, `Z002432Wide`/`Z002433Wide`. `GenerateShifts` builds a 3-shift calendar table for these.
**Auth tables that are not the Manager's:**`UserAuth`, `UserAuthDetails`, `UserAuthRole`, `ApiKey` are ServiceStack's schema — they belong to **MESAPI/WWSupport** (the Camstar-facing REST service, see `mes-delmia-integration-api.md` §1.1: `ApiKeyAuthProvider` + `LdapAuthProvider`, role `MESAPI`), which shares the `BT` database. The Manager authenticates via LDAP only.
**Notification tables** (`Notification`, `NotificationLog`, audited by trigger) back the separate `WWNotificationSystem` Windows service, which is **not in this repo**.
## 4. BTDBManager (the config/edit UI) — `master` state; the replacement is §8
- **Auth:** cookie scheme; `LdapAuthenticationService` binds `LDAP://nam.zimmer.com` with the user's own credentials via `System.DirectoryServices` (so it only runs on a domain-joined Windows host), reads `memberOf`, and keeps only the three configured group DNs. Policies `READONLY` / `MACHINECONFIG` / `BATCHEDIT` map to `ReadOnlySecurityGroup` / `MachineConfigSecurityGroup` / `BatchEditSecurityGroup`; the checked-in DEV config points all three at `UG_WW_APP_ADMIN_DEV`. A global `AuthorizeFilter` makes every controller (incl. `/Api/GetMachines`, `/Api/GetMachineGroups`) require login. This is exactly the AD-group→role problem the family's `ZB.MOM.WW.Auth` 0.3.0 + `ldap_mapping.md` already solve.
- **Features:** Machine CRUD (with group memberships + alarm catalog via TVPs `MachineGroupMembershipType`/`MachineAlarmType`), MachineGroup CRUD, batch search/edit (`SaveBatchEdit` writes `AuditLogBatchEdit` + prev/new `AuditLogMachineBatch`/`…WO`/`…Cycle` rows), **MachineConfigImport** (copies machine/group config from another environment's `BT` — sources `QA` and `PROD` are hard-wired in `appsettings.json`), and Telerik `.trdp` audit reports.
- **Secrets are checked in:** `appsettings.json` carries plaintext `wonderapp` SQL passwords for DEV/QA/PROD, and `install.bat` carries a domain service-account password. Treat all of them as compromised-by-repo; do not copy them into any other file or reply. This is the pattern `ZB.MOM.WW.Secrets` (`${secret:}`) exists to replace.
- **Environment hosts as recorded in 2023:** DEV `wonder-sql-vd01/BT`, QA `wondersqlqa/BT` (`MultiSubnetFailover`), PROD `wondersqlpd/BT`. ScadaBridge's 2026-06-16 design says the DEV BTDB is on **`wonder-sql-vd03/BT`** (db-connection id 1). `mes-delmia-integration-api.md` also says `vd01`. **Host drift is unresolved** — do not assume either without probing.
- **Galaxy → BT** is the primary write path (batches/cycles/work orders/OtherData JSON). DARS `model_info.md` §: "Existing per-machine BT sprocs (`StartMachineBatch`, `StartMachineCycle`, `GetCycleTSV`, `Alarm_Wrapper`, …) work unchanged" — the sprocs are the contract Galaxy scripts bind to.
- **ScadaBridge → BT (verified):** the inbound-API script context has a `Database` helper bound to named db-connections; production defines connection **`BTDB`** (id 1), and the `IpsenMESMoveIn` (design 2026-06-16) and MES alarm-status (2026-06-30) methods do `SELECT Code FROM dbo.Machine WHERE SAPID=@s` before `Route.To(instance)`. Template/site scripts expose `Database.Connection(name)` + `Database.CachedWrite(...)`, so site scripts **can** write batches into BT — but **no checked-in ScadaBridge script calls a BT sproc**; any batch logging is customer-site script configuration, not platform code, and the docker rig does **not** seed a `BT` database (`infra/mssql/setup.sql` creates only `ScadaBridgeConfig` + `ScadaBridgeMachineData`; the `InboundDatabaseHelperTests` use a stand-in). ScadaBridge's inbound API is the designated successor of MESAPI's move-in/move-out surface, which is why it needs the same `SAPID → Code` lookup.
- **HistorianGateway ↔ BT:** no direct coupling, but both read `Runtime.dbo.Events` (HG's SQL-path `ReadEvents`) and both need the Historian SQL instance. BT's CLR `GetTSV`/`GetAggregateValues` is a bespoke, in-database equivalent of what HG's `ReadRaw`/aggregate retrieval modes do over gRPC — a candidate for consolidation if the report layer is ever re-platformed.
- **OtOpcUa / mxaccessgw:** no coupling.
- **Break surface:** the sproc signatures + `Machine.Code`/`SAPID`/`ZTag` semantics (Galaxy scripts, MESAPI, ScadaBridge inbound methods all bind to them), the `OtherData` JSON key conventions (DARS sprint 1 defines ~14 new keys with single writers), and the `INSQL`/`Runtime` co-location assumption.
## 6. Analysis — gaps and recommendations (2026-09-03)
1.**Push the branch.**~~Put both copies under git~~ — done locally 2026-09-03 (`feat/machine-batch-transfer`). Remaining: push it to `ZimmerBiomet/BTDB`, review/merge, decide whether BTDB also gets a mirror on `gitea.dohertylan.com` like the other sisters, then delete the superseded `DARS/WW_ARCH/BatchTracker_DB/` copy.
2.**Rotate and externalize the checked-in credentials** (three SQL passwords, one domain account). _Externalised on `feat/manager-blazor` (`${secret:btdb/...}` refs, store outside the content root); rotation is a cut-over step (§8)._ The family standard is `ZB.MOM.WW.Secrets``${secret:}` with an absolute `Secrets:SqlitePath` outside the content root.
3.**Manager re-platform** — _done on `feat/manager-blazor`, see §8_: net6.0 is EOL; Telerik is pinned by an absolute machine path; `System.DirectoryServices` binds the user's own credentials on a domain-joined box. A .NET 10 rewrite on the family libs is mostly assembly: Auth 0.3.0 (LDAP + `UG_WW_*` group→role mapping, `FallbackServers`), Theme 0.4.1, Health 0.3.0 (`MapZbHealth`, so Overview can watch it), Telemetry, Configuration, Audit (the batch-edit audit is a natural `ZB.MOM.WW.Audit` writer), Secrets. Keep Dapper + the embedded queries; replace Telerik `.trdp` with server-side rendering only if the three audit reports are still used.
4.**Injection surfaces in the SQL layer:**`AddOtherData*` build dynamic SQL from `@DataKey`/`@DataValue`; `GetTSV`/`GetAggregateValues` interpolate tag names and append `@queryParams` raw into `OPENQUERY`. Trusted callers today, but ScadaBridge scripts and the inbound API widen the caller set. Fix `AddOtherData*` with parameterised `JSON_MODIFY` (no dynamic SQL needed), and whitelist `queryParams` to the three `ww*` keys.
5.**ServiceStack auth tables** (`UserAuth*`, `ApiKey`) are MESAPI's, not BT's domain. Once ScadaBridge's inbound API fully replaces MESAPI (Ipsen MoveIn is the first slice; ScadaBridge keys already live in the `zbauth` schema via `Auth.ApiKeys.SqlServer` 0.3.0), those tables are dead weight in the schema and should be dropped from the sqlproj rather than carried forward.
6.**Host inventory drift:** 2023 config says DEV BT is on `wonder-sql-vd01`; ScadaBridge's 2026 design says `wonder-sql-vd03`. Probe and record the answer in `deployment.md` §2.1 (which currently has no BT row at all).
7.**`install.bat`/`uninstall.bat` are wrong for this project** (they install `WWNotificationSystem`). _Deleted on `feat/manager-blazor`; replaced by `deploy/install-service.bat` (NSSM, service `ZB.MOM.WW.BTDB.Manager`)._
8.**CLR + Historian co-location is a hard deployment constraint** the index did not previously record: BT cannot be moved to the app-tier SQL (`wonder-app-vd03,1433`, where ScadaBridge's ConfigDb lives) without breaking every report wrapper. Any consolidation plan must keep BT on the Historian instance or replace the CLR with HistorianGateway calls.
## 7. Primary commands (Windows only)
```powershell
# from ~/Desktop/btdb (master)
dotnetbuildManager\BTDBManager.csproj# net6.0; needs Telerik Reporting R2 2022 at the HintPath
docker/deploy-db.sh # BT on scadabridge-mssql from Database/ (dacpac + SQL CLR); --recreate drops it first
docker/deploy.sh # publish FDD on the host → zb-btdb-manager:local, up on :9877 (glauth login enabled)
```
On `master` nothing builds on macOS (SSDT/CLR need Windows MSBuild, the MVC Manager needs Telerik + `System.DirectoryServices`). On the branch the app and its tests build and run on macOS; only `Database.sqlproj` (kept alone in `BTDB.sln`) still needs Windows — but §9's side projects build the same sources into a dacpac and a CLR assembly anywhere, for the local rig.
Plan: `btdb/docs/plans/manager_blazor_conversion.md` (+ `.tasks.json` tracker beside it; mockups `docs/mockups/manager_blazor_mockups.html`). Executed 2026-09-04 in worktree `~/.config/superpowers/worktrees/BTDB/feat-manager-blazor`; **uncommitted and unpushed at the time of writing** — phases 1–3 (scaffold, core, auth, widgets, queries, all pages) plus the local half of phase 4 are done; the DEV-box half of phase 4 (parity run, secrets seed + rotation, NSSM side-by-side, port swap) is written up as a runbook and not yet performed.
- **Shape on the branch:** `BTDB.slnx` = `src/ZB.MOM.WW.BTDB.Manager` (Blazor Web App, InteractiveServer, net10.0) + `src/ZB.MOM.WW.BTDB.Manager.Core` (Dapper data layer moved verbatim from the MVC app, hardened) + two xunit/bUnit test projects (334 tests, warnings-as-errors gate via root `Directory.Build.props`, `build.sh`/`build.ps1`). `BTDB.sln` keeps only `Database.sqlproj`. `Manager/` (MVC, Kendo, the three Telerik `.trdp` reports with embedded connection strings, the mis-named `install.bat`) is **deleted**; `/Api/GetMachines`+`GetMachineGroups` are gone (no external consumer — ScadaBridge/MESAPI read `dbo.Machine` by SQL).
- **Family libs:** Theme 0.4.1, Health 0.3.0 (`MapZbHealth`; readiness = a `SELECT 1` against `BT` with a 3 s timeout, so `/health/ready` is 503 until the DB answers — register it in Overview), Telemetry + Serilog 0.1.0, Configuration 0.1.0 (`ConfigPreflight` requires `ConnectionStrings:BT` and `Btdb:Security:Ldap:Server`), Secrets 0.6.2 (`${secret:btdb/connectionstrings/{bt,qa,prod}}`, `${secret:btdb/ldap/serviceaccountpassword}`; SQLite store `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`, KEK from `ZB_SECRETS_MASTER_KEY`), Auth.Ldap/AspNetCore 0.3.0 (service-account bind, `Btdb:Security:GroupToRole` → `CanonicalRole`; policies Viewer / BatchEdit (Operator+) / MachineConfig (Administrator); cookie `btdb.auth`, plain-HTTP cookie because the box has no certificate). `Microsoft.Data.SqlClient` 7.x with `Encrypt=False` forced by `BtConnectionStrings.Normalize` unless the string sets it.
- **Behavioural deltas vs the MVC app worth knowing:** config import derives the group set from the selected machines' memberships (no separate group pick) and has a dry-run preview (`MachineConfigImportPreview.sql`: same MERGEs, `OUTPUT` into a table variable, always `ROLLBACK`); batch-edit timestamps are 7-fraction UTC strings validated on blur (no Kendo mask); the three audit reports are HTML tables with a print stylesheet instead of Telerik PDFs; `MachineAlarm` DTO property order/types were wrong in the MVC app (alarm import always threw) and are fixed against `MachineAlarmType`; every `Save*` runs in a transaction and `THROW 5001x` maps to `BtdbSaveException`.
- **Cut-over material:** `docs/operations/cut-over.md` (parity checklist per route + THROW paths + audit-username check, secrets seed via the `secret` console + rotation, NSSM side-by-side on `:9877`, port swap to `:9876`, Overview registry entry), `deploy/` (`publish.ps1`, `install-service.bat` / `uninstall-service.bat` for NSSM service `ZB.MOM.WW.BTDB.Manager`), `tools/smoke-new-queries.sql` (runs the six never-executed queries read-only against DEV with sqlcmd). Still open after cut-over: Overview lacks the mockup's per-machine "open > 24 h" rows; Ldap `Server`/`SearchBase`/`ServiceAccountDn` are blank in `appsettings.json` by design (per-environment).
## 9. Local Docker rig — `docker/` (2026-09-04)
Mirrors the HistorianGateway / Overview pattern: publish framework-dependent on the host (Gitea feed creds), copy into `aspnet:10.0`, never restore in Docker. Compose project `zb-btdb`, container `zb-btdb-manager`, image `zb-btdb-manager:local`, port `:9877` (same as the DEV side-by-side port, so `appsettings.json`'s Kestrel section is the single binding authority; the Dockerfile clears `ASPNETCORE_URLS`/`ASPNETCORE_HTTP_PORTS`). `ASPNETCORE_ENVIRONMENT=Production`; all environment-specific values are compose env vars, dev-only and committed under the family's rig exception.
- **Database:** no MSSQL of its own — joins the external `scadabridge-net` and uses ScadaBridge's `scadabridge-mssql` (the instance that already carries `ScadaBridgeConfig` + `ZbSecretsHub`). `docker/deploy-db.sh`: (1) `docker/db/BT.Clr.csproj` compiles `Database/**/*.cs` to `Database.dll` (net472 via the auto-referenced reference assemblies — builds on macOS); (2) `docker/db/BT.Local.sqlproj` (SDK-style `Microsoft.Build.Sql/2.2.0`, `EnableDefaultSqlItems=false`, globs the four `Database/` folders, Sql130 model, suppresses the legacy build's own SQL71562/71558/71502) → `BT.Local.dacpac`; (3) `db/setup.sql` — `BT` + login `btdb_app` (db_owner); (4) `db/clr-objects.sql` — `clr enabled` on the instance, `sp_add_trusted_assembly` by SHA2-512 (strict security stays on), `CREATE ASSEMBLY [Database] … SAFE` from the bytes streamed as 2000-char `0x` chunks (a single 57 KB literal gets split by sqlcmd), then `StringJoin` aggregate, `StringCompare`/`CheckTriggered`/`WhenTriggered`/`WhenCleared`/`GetAggregateValues`, `GetTSV`/`GetMachineBatchTSV`/`GetCycleTSV`/`GetLinkedBatchTSV` with SSDT's type mapping; (5) `sqlpackage /Action:Publish` from a one-shot container (`db/Dockerfile.sqlpackage`, SDK image + the dotnet tool — the host's 10.0.5 runtime could not run the tool's 10.0.11 pin) with `DropObjectsNotInSource=false`, `BlockOnPossibleDataLoss=false`, `AllowIncompatiblePlatform=true`; (6) smoke. Result on the rig: 23 tables, 37 T-SQL procs + 1 TVF, 10 CLR objects, `Database` assembly `SAFE_ACCESS`. The script diffs the legacy `<Build Include>` list against the folders and warns on drift (found: `ElmaData_Nitric.sql` + `Elma_Bulk_Query_Nitric.sql` were on disk but not in `Database.sqlproj` — added). `ScadaBridge/infra/teardown.sh`/`reseed.sh` drop the volume and BT with it; re-run `deploy-db.sh`.
- **Why not convert `Database.sqlproj` itself:** `Microsoft.Build.Sql` does not compile SQL CLR, and the Manager depends on `dbo.StringJoin` (batch search) and `dbo.StringCompare` (machine saves). The legacy SSDT project stays the artifact of record; the two side projects reference the same files, nothing is copied.
- **Login:** enabled, `host.docker.internal:3893` + `FallbackServers``:3894` (glauth-redundant), bind `cn=serviceaccount,dc=zb,dc=local`; `GroupToRole``SCADA-Admins→Administrator`, `SCADA-Designers→Operator`, `SCADA-Viewers→Viewer`. Verified: `admin`/`designer` sign in, `gw-viewer` refused (fail closed). Literal `ConnectionStrings__BT` / LDAP password env vars override the `${secret:}` refs, so the `/data/btdb-secrets.db` store stays empty and no master key is set (confirmed to boot without one). The two config-import sources point at the local BT.
- **Health / Overview:** `/healthz` 200, `/health/ready` 200 with the `sql` entry (503 when BT is down), `/health/active` 200, `/metrics`. Compose `healthcheck` = bash `/dev/tcp` GET of `/healthz` (liveness on purpose). Registered in `ZB.MOM.WW.Overview/src/…/appsettings.Docker.json` as application **BTDB Manager**, instance `manager`, `BaseUrl http://zb-btdb-manager:9877`, `HasActiveRole false`; the containerised Overview was republished and shows the tile Up with `sql` healthy (the dashboard already sits on `scadabridge-net`, so no extra network).
## 10. State as of 2026-09-11 (supersedes §1, §4, §6.1, §6.3, §6.7, §8 where they differ)
- **Everything is on local `master`** (`78195b8`): the Blazor conversion (§8) was merged, the
`feat/manager-blazor` worktree is gone, and the 2026-09-04 full review
(`docs/reviews/2026-09-04-full-review.md`, reviewed at `c624d41`) was remediated in 12 work-package
- **Invariants a future session must not break** (all test-enforced): TVPs bind by ordinal so model
property order + lengths must equal `Database/Types/*.sql`; every write query is transaction-free and
audited (the C# wrapper owns the transaction); `dbo.X` only, never `BT.dbo.X`; `SYSUTCDATETIME()` in
Manager queries; every writer sproc follows the own-or-join `TRY/CATCH` template with locked lookups;
every `.sql` on disk must be in `Database.sqlproj`'s `<Build Include>` list; optimistic concurrency via
`ExpectedLastUpdate` → `THROW 50014`.
- **Secrets/credentials (§6.2):** `${secret:btdb/connectionstrings/{bt,qa,prod}}` (the LDAP
service-account secret is gone since 2026-10-01 — see below; only `dotnet run` against local glauth uses it), store at `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`,
master key only in `ZB_SECRETS_MASTER_KEY`. Rotation is cut-over step 4.2.
- **Claude artifacts are gitignored + untracked in btdb** (2026-09-11): `CLAUDE.md`, `.claude/`,
`docs/plans/`, `docs/reviews/`, `*.tasks.json` — they exist on disk (btdb's `CLAUDE.md` is the repo
orientation and still loads) but never reach the corporate remote. `docs/operations/`,
- **2026-10-13 existing fixes verified and issues closed.** Main `513fa1aa` already includes #35 fail-closed unavailable configured body/SQL redactors (`006202f3`) and #37 Flow Debugger structured/raw validated method inputs (`b5eb08f1`). Fresh full AuditLog project 407 passed/0 failed; FlowDebuggerPageTests 47 passed/0 failed; both terminal exit0. Remote ancestry confirmed; #35/#37 closed with evidence. No source changes/deployment/MES transactions. #36 runtime reset-echo overload is implemented, but QA/prod script port remains tracked; #30 per-site PSK deployment gate remains open.
- **2026-10-13 #36 runtime regression proof refreshed.** Rebuilt HandshakeResetEchoTests passed all three cases with local fakes, terminal exit0; an earlier no-build attempt matched no tests and is excluded from evidence. Tracker now distinguishes implemented runtime b623fdb0 from remaining QA/prod database-owned script rollout (deployment register row119, scripts13/14/36 and15/16/25/26/37/38). No database writes, deployments or fresh live rig claims.
- **2026-11-13 deployed `8e32f353` to wonder-app-vd03 (DEV) and enabled the audit hash chain** (was `424db155`; owner go on #107). 37 commits, no migration (64). Build: clean worktree, self-contained win-x64, zip SHA-256 `0cf06cf5…7fdebc`, LocalDb 0.3.8. WinRM swap 07:13–07:14 EST, site data path ~45 s, central API ~25 s; `sc failureflag 1` on both services (restart actions existed, but non-crash failures were FALSE, so a #132 exit 75 would have left the node stopped). Env block digest-identical before/after. **#131 applied** (`apply.sh apply 131`, SBTEST01/02 deployments 95/96; verify groups C+D 13/13 on the old binary and again after; C13 skipped because SAPID 103559 routes to Z28061) — #131 closed. Gates: health 200 ×4, MxGateway Connected 398/412 = baseline, Z28062 144 Good = baseline, AuditLog writing, Overview Up. **Hash chain:** backfill 843 → `unknown`, then `AuditLog__HashChain__Enabled=true` in the Central service env (ChainBacklog unset, head export off, single login), Central-only restart; one anchor (2026-11 `ChainSeq 0`, cutoff 2026-11-13T12:16:53.114Z), 0 pre-cutoff rows chained, `verify-chain 2026-11` Intact with 507 pre-cutoff rows, 2026-10 NotChained (667). Backup `ScadaBridge_pre-8e32f353_20261113T071302.bak`; rollback scripts in `E:\ApiInstall\_stage-sb-8e32f353\`. Record `docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md` (`581e25ca`). QA not deployed.
- **2026-11-11 deployed `424db155` to wonder-app-vd03 (DEV)** (was `0e540291`). Owner-approved WinRM deploy. Migrations 58→64 (`AddNativeAlarmSourceMesReportable`, `PrefixSearchIndexes`, `AddAuditLogHashChain`, `AddConfigGraphRowVersions`, `FlipAuditLogClusteredKey`, `AuditMaintenanceGrants`) applied by script with central + site stopped; #111 DEV sizing: 2,394 rows / 8 MB flip in ~1.5 s. **The plain `dotnet ef migrations script --idempotent` output fails (Msg 207 `ChainSeq`: column added and indexed in one batch)** — the deployed script adds `GO` per guarded block + `XACT_ABORT`; forward/rollback rehearsed on a restored copy of the vd03 DB. Downtime ~50 s; verify OK. Backup `ScadaBridge-backups\ScadaBridge_pre-424db155_20261111T164219.bak`. QA not deployed.
- **2026-11-12 #129 alarm Severity runs one direction (higher = more urgent), main `dd4f1c2a`.** MxGateway native alarms and computed `PriorityLevel` now band AVEVA-style priority onto `AlarmConditionState.Severity` with OtOpcUa's `GalaxyAlarmPriority` table (lmxopcua#887): 1–250→1000, 251–500→750, 501–750→500, 751–999→250, ≤0→1000, >999→250 (`AlarmPriorityBands`). OPC UA rows are unchanged. The raw number (= pre-#129 Severity) is kept on new `AlarmConditionState.Priority`, `ScriptAlarm.Priority` and `AlarmStateChanged.Priority`. There is no proto change: central derives Severity from the wire `priority` by kind, and pre-#129`native_alarm_state` rows are upgraded on rehydrate. The native MES 900–999 band and the MES API `Severity`/`MinSeverity`/`MaxSeverity` key off `Priority`, so they select the same alarms as before. The rig seed scripts were switched; **deployed MES scripts (CvdReactor, vd03 SBTEST01/02) must switch `a.Severity`→`a.Priority` in the same deploy: deploy gate ScadaBridge#131.****Visible flip:** Alarm Summary worst/filter/sort and the Sev column now rank AVEVA priority 1 highest. A computed raise logs `Error` at priority 1–500 (was raw ≥700). Tests (Release): Commons 977, DCL 549, Communication 834, SiteRuntime 1055, CentralUI 1545, all green; a mutation revert turned 62 red. Not deployed.
- **2026-11-12 OPC Foundation SDK 1.5.378.106 → 1.5.378.182 (`main``e36cd605`, ScadaBridge#125 ask 2, owner option B; not deployed).**
- Client (DCL) and the test-only Server move in lockstep. No source changes, no API breaks.
- Release build has 0 warnings. DCL tests pass 536/536.
- **Wire-compat re-run** with the bumped client, using the lmxopcua harness (`438484fd`, run at master `2e9cccbd`), ACL off:
- The matrix passes 11/11 in None, Sign and SignAndEncrypt, against both the SDK 1.5 baseline `83ed880e` and the SDK **2.0.1** master `a28adba2`.
- All 8 trust controls fail closed (0x80130000 and 0x801A0000).
- Failover passes 8/8 legs, with the same timings as on the old client.
- **Operational note:** since .156, a CA-issued OPC UA server cert that the DCL trusts needs an issuer with `keyCertSign`. SDK self-signed certs are unaffected.
- **2.0.x is deferred** until lmxopcua#799 and #801 close. #125 stays open to track it.
- **2026-11-12 OPC Foundation SDK 1.5.378.182 → 2.0.1 (`main``a2e9d718`, ScadaBridge#125 closed; not deployed).** Done once lmxopcua#782/#799/#801 had closed. Client (DCL) and the test-only Server move in lockstep; the transitive Core/Core.Types/Types/Configuration/Security.Certificates packages resolve to 2.0.1.
- **API breaks fixed (DCL only):**
- Immutable `Variant`/`DataValue`/`NodeId`/`StatusCode` and `ArrayOf<T>`. New `OpcUaValueBoundary` restores the SDK 1.5 CLR value shapes (`byte[]`, UTC `DateTime` with no-timestamp = `MinValue`, `Guid`, detached arrays) and keeps `new Variant(object)` write semantics.
-`StatusCodes` are no longer constants, so code matches them by value.
-`CertificateIdentifier.Certificate` and the `CertificateValidation` event are gone. The SDK-created `CertificateManager` owns the identity and trust lists and is disposed with the session; auto-accept and the verify probe use `AcceptError`.
- X509 user identity now goes through `X509IdentityTokenHandler`.
- **UserName = `LegacyRsaUserNameIdentityTokenHandler`** (ported from OtOpcUa `be28aa44`, lmxopcua#880). SDK 2's default handler sends a 65-char API key as an `RsaEncryptedSecret`, which SDK 1.5 servers reject (`BadIdentityTokenInvalid`) and SDK 2 servers clock-gate (#873). The DCL sends what it sent on 1.5, so it works against both server generations in a mixed rollout.
- **Operational:** SDK 2 never accepts an incomplete chain or an empty subject/issuer DN, even with `AutoAcceptUntrustedCerts`. The `keyCertSign` rule from .156 still applies.
- **Tests:** Release 0 warnings; DCL 570/570 incl. the #121 secure-connect tests and opc-plc live tests. The new handler tests are mutation-checked. Whole solution 11,345 passed / 188 failed (185 Playwright with no browser server, 3 load flakes that pass in isolation).
- **Wire-compat re-run:** lmxopcua harness, its key now the 65-char production shape (`2705f166`). Against baseline `83ed880e` (SDK 1.5) and master `412da27c` (SDK 2.0.1):
- Matrix 11/11 ×3 modes on both servers, ACL off and on; 8 trust controls fail closed.
- Failover 8/8. Kill reaches the backup in about 20 s as before; freeze is detected in about 15 s (was about 20 s).
- A mutation run with the SDK default UserName handler fails every SDK 1.5 leg with `BadIdentityTokenInvalid`, which is #880 reproduced from the consumer side.
- Evidence: `~/validation-evidence/scadabridge-125/client-2.0.1/` (contains private PKI; not committed).
Shared audit event model + writer seam for the **ZB.MOM.WW SCADA family** (single package; DEEP-adopted by all apps as the canonical record). Build/test from this directory: `dotnet test`. Component spec + gaps: `../components/audit/`.
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
### Narrative
The audit component is fully populated: a normalized [`spec`](components/audit/spec/SPEC.md), an
[`event-model`](components/audit/spec/EVENT-MODEL.md) reference, a
[`shared-contract`](components/audit/shared-contract/ZB.MOM.WW.Audit.md), three
[`current-state`](components/audit/current-state/) docs, and an adoption [`GAPS`](components/audit/GAPS.md)
backlog. Common ground = canonical `AuditEvent` record + `AuditOutcome` enum + `IAuditWriter` /
Adoption is tracked in `components/audit/GAPS.md` in the outer `scadaproj` workspace. Each app brings its own transport (Akka broadcast / SQLite append / SQL ingest) and domain vocabulary (channels / kinds / event-types) — those stay per-project. The shared library provides the canonical record and the two seams that decouple "what to audit" from "how to store it".
All four apps pin `0.1.0` (the only version on the Gitea feed), verified against each repo's origin default
branch 2026-11-09. The 2026-06-02 adoption was merged and pushed on 2026-06-03.
Adoption history is in `components/audit/GAPS.md` in the outer `scadaproj` workspace. Each app brings its own transport (Akka broadcast / SQLite append / SQL ingest) and domain vocabulary (channels / kinds / event-types) — those stay per-project. The shared library provides the canonical record and the two seams that decouple "what to audit" from "how to store it".
---
@@ -53,7 +57,7 @@ dotnet test ZB.MOM.WW.Audit.slnx
Full design docs live in the `components/audit` folder of the SCADA project workspace:
Shared login / identity / API-key library for the **ZB.MOM.WW SCADA family** (4 packages + `ApiKeys.SqlServer`; consumed by OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). Build/test from this directory: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`). Component spec + gaps: `../components/auth/`.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #5 (roll apps to `0.4.1`), #6 (VD03 NAM AD cutover), #18 (Secrets migrator jitter, ported from here). The history below is a record, not a status tracker; file new follow-ups as issues.
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
| Auth (login / identity / authz) | Adopted (**feed carries `0.3.0` as of 2026-09-02 — five packages, the new one being `ZB.MOM.WW.Auth.ApiKeys.SqlServer`; ALL FOUR apps pin `0.3.0` as of 2026-09-03**: OtOpcUa `master @ b8b4c18f`, ScadaBridge `main @ e45c4af9` (the `inbound-api-key-shared-store` work, merged), MxAccessGateway `main @ 4f931e4`, HistorianGateway `main @ d8d890c` — each built + tested with no new failures versus its pre-bump baseline, all PUSHED to origin. The preceding `0.2.1` pins landed 2026-08-13 (OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`, MxAccessGateway `1cb14d2`, HistorianGateway `e42ff31`). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
### Narrative
The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
proposed [`shared-contract`](components/auth/shared-contract/ZB.MOM.WW.Auth.md), three
[`current-state`](components/auth/current-state/) docs, and an adoption [`GAPS`](components/auth/GAPS.md)
backlog. Common ground = LDAP/GLAuth identity + peppered-HMAC API keys; left per-project =
**Adopted across all three apps on 2026-06-02** (auth GAPS #1–#8) on each repo's `feat/adopt-zb-auth` branch —
committed + reviewed, then **fast-forward-merged into the repo's local default (main/master) and PUSHED to origin
(gitea) on 2026-06-03** (in sync; the `feat/*` branches kept locally as history). Cutover: shared `Auth.Ldap`,
`Auth.ApiKeys` (ScadaBridge inbound fully re-architected to the keyId/Bearer model), `IGroupRoleMapper<TRole>` seam,
`Transport`-enum config, canonical `ZbClaimTypes`/`ZbCookieDefaults`, unified dev base DN `dc=zb,dc=local`, and the
canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins: **all four apps pin `0.3.0` as of 2026-09-03** (commits listed in the table row above, all pushed to origin); all four pinned `0.2.1` from 2026-08-13 before that. Prior history — all four pinned `0.1.5` (verified 2026-08-07): MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
Build/test from `ZB.MOM.WW.Auth/`: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`, skipped otherwise). Consumer matrix: OtOpcUa → Abstractions+Ldap+AspNetCore;
MxAccessGateway & HistorianGateway → the original four; ScadaBridge → those four **plus `ApiKeys.SqlServer`** (ApiKeys not used by OtOpcUa).
**`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger<LdapAuthService>` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up.
**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.****Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`).
**`0.3.0` (2026-09-02) — `ZB.MOM.WW.Auth.ApiKeys.SqlServer`, a SQL Server provider for the API-key store** (design `ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; **5 pkgs published to the feed**, `ApiKeys.SqlServer` search-verified from the ScadaBridge checkout; family `Directory.Build.props` now `0.3.0`). Motivation: `Auth.ApiKeys`' SQLite store is a per-node file, so on a two-node active/standby host (ScadaBridge central) a key minted on one node exists only there and a failover 401s every inbound caller (ScadaBridge deferred-work row 108). The new package implements `IApiKeyStore`/`IApiKeyAdminStore`/`IApiKeyAuditStore` over `Microsoft.Data.SqlClient` in a `zbauth` schema (v1: `schema_version`, `api_keys`, `api_key_audit`; all timestamps `DATETIMEOFFSET(7)`, tick-exact) with a concurrent-boot-safe migrator (Serializable tx, retry on 1205/2714/1913/2627 with **jittered** backoff — the Secrets replicator's flat `100*attempt` backoff exhausted five retries about once per sixty 8-way boots in the port's soak test, so the same latent defect is an open follow-up there → #18) and a `SqliteApiKeyStoreImporter` that copies an existing SQLite file into the schema insert-where-absent (idempotent, so both nodes of a pair can import their own file and the result is the union). Wiring: `services.AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)` registers the SQL Server stores + migrator **before** delegating to `AddZbApiKeyAuth`, whose `TryAdd*` registrations then yield (same displacement mechanism as Secrets 0.5.0). One `Abstractions` addition, additive: `IApiKeyStoreMigrator`, which `SqliteAuthStoreMigrator` now implements and `ApiKeyAdminCommands` accepts via a new ctor (old `SqliteAuthStoreMigrator` ctor retained — the three `0.2.1` consumers compile unchanged). Store semantics are pinned to the SQLite contract tests re-run live (mark-used skips revoked keys, delete only removes revoked keys, rotate clears last-used/revoked, list newest-first). 92 tests in the new project, 351 across the solution, 0 failed. A DBA-run copy of the DDL ships at `docs/sql/zbauth-schema.sql`.
**`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real**`LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups '<pat>'` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain.
**`0.5.0` (2026-11-12, ScadaBridge#46) — passwordless group lookup; first published version carrying 0.4.2–0.4.4.** New `Abstractions` contract `ILdapGroupLookup.GetUserGroupsAsync(username, ct)` → `LdapGroupLookupResult` / `LdapGroupLookupStatus` (`Found`, `UserNotFound`, `UserDisabled` (AD `userAccountControl` bit 2), `AmbiguousUser`, `DirectoryUnavailable`, `NotSupported` (`BindMode=User` / blank service DN / foreign `ILdapAuthService`), `ProviderDisabled`). Implemented by `LdapAuthService` (service-account bind + `UserNameAttribute` search; same extraction/canonicalisation as sign-in; the endpoint walk was factored into one generic `WalkEndpoints<T>` shared by both paths, so failover and the sticky preference are common). `AddZbLdapAuth` registers it as the same singleton. Tests: 30 unit (`LdapGroupLookupTests`, 6 mutations all killed) + 2 DI + 6 live GLAuth (`GLAuthGroupLookupTests`, run against the redundant local pair incl. two-live-endpoint failover; failover mutation killed live). **scadaproj#51 closed by a live NAM run (24/24)** — see README §0.5.0/§0.4.4 for what was proven; the identity check's non-vacuity was shown by mutating the `msDS-PrincipalName` comparison (user-bind sign-ins then fail). Consumers: ScadaBridge adopts 0.5.0 (#46); OtOpcUa / mxgw / HistorianGateway / BTDB stay on their pins.
**`0.4.4` (first published in 0.5.0, scadaproj#40, Auth-007):** `BindMode=User` now proves the entry whose groups it returns is the account that bound — RFC 4532 Who-am-I (`dn:` → DN match; AD `u:DOMAIN\name` → `sAMAccountName` match), falling back to the bind DN only for a DN template (GLAuth has no Who-am-I); otherwise `GroupLookupFailed`. `{0}` is RFC 4514-escaped in DN templates. Before the fix a user could sign in with another account's groups whenever the UPN/DN bind and the `UserNameAttribute` search named different accounts — reproduced live against a throwaway GLAuth (`tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/glauth-userbind/`, `ZB_LDAP_USERBIND_IT=1`, port 3897): pre-fix `mallory` signed in with `Admins`. **Consumer impact:** AD + down-level/UPN template unchanged for consistent accounts (BTDB Manager is the only user-bind consumer); non-AD user-bind needs a DN template that yields the entry's DN exactly as searched (GLAuth: `cn={0},ou=<group>,ou=users,<base>`). README §0.4.4.
**`0.4.4`, continued (first published in 0.5.0, scadaproj#46 + #50):** still `0.4.4`.
- **Auth-008 (`ApiKeys.SqlServer`):** over-length key records are refused with `ArgumentException`; an over-64 key-id lookup answers not-found without querying; every `key_id` predicate adds an exact `VARBINARY` match, so lookups are case- and trailing-space-exact like SQLite. No schema change. `ApiKeyImportResult` gains the `init` lists `KeyIdCollisions` and `KeysRejected`, and `KeysSkipped` now counts exact duplicates only. Audit text is still truncated to 64 characters, deliberately (documented).
- **Auth-009:** the canonical username is the `UserNameAttribute` value matching the typed name, ignoring case, not `values[0]`.
- **Auth-010:** the new `ApiKeyFailure.StoreUnavailable` (appended) is returned when `FindByKeyIdAsync` throws; only cancellation still throws.
- **#50 (user bind, AD forests):** for a `u:DOMAIN\name` Who-am-I answer, the entry must be that domain's account. Where AD offers `msDS-PrincipalName` (a base read through the new internal `ILdapConnection.Read`), it must equal the answer. Otherwise the name must have exactly one holder under `SearchBase`. Live-verified against the NAM AD 2026-11-12 (scadaproj#51, see 0.5.0).
- **Consumer-facing:** a new enum value; SQL Server key ids now match exactly; over-length creates throw; the importer reports collisions and rejections.
- Suites: 363 passed / 81 skipped offline; 440 passed / 4 skipped (LDAP live) with `AUTH_SQLSERVER_CONNSTR`, against a throwaway SQL Server 2022.
**`0.4.3` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). No API change.
**`0.4.2` (unpublished, scadaproj#23 + #22):** `ZB.MOM.WW.Auth.ApiKeys.ApiKeyPepperUnavailableException : InvalidOperationException` (message still `"pepper unavailable"`) is now thrown by `ApiKeyAdminCommands` create/rotate when the pepper is missing, so consumers can catch it by type; existing `catch (InvalidOperationException)` blocks keep working (only exact-type assertions like xunit `ThrowsAsync<InvalidOperationException>` need updating). **Release note: consumers can drop their own `IApiKeyPepperProvider` pre-checks** — mxaccessgw's `DashboardApiKeyManagementService` and any similar code. Also pins `System.Security.Cryptography.Pkcs 9.0.21` (transitive, via SqlClient 6.0.2) to clear NuGetAudit NU1902/NU1903. Not yet published to the feed.
@@ -11,17 +11,18 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
| `ZB.MOM.WW.Auth.Abstractions` | Auth contracts, canonical role constants, and shared types (`LdapOptions`, `LdapAuthResult`, `ILdapAuthService`, `IApiKeyStore`). No runtime dependencies beyond the BCL. | — |
| `ZB.MOM.WW.Auth.Ldap` | LDAP authentication service: bind-then-search-then-bind against GLAuth or Active Directory; RFC 4514-aware group extraction; fail-closed. | `Abstractions`, `Novell.Directory.Ldap.NETStandard` |
| `ZB.MOM.WW.Auth.ApiKeys` | SQLite-backed API-key store with **pepper-keyed HMAC-SHA256** secret hashing, rotation, and audit log. DI wiring is `AddZbApiKeyAuth`; an opt-in `MigrationHostedService` runs schema migrations on startup. | `Abstractions`, `Microsoft.Data.Sqlite` |
| `ZB.MOM.WW.Auth.ApiKeys.SqlServer` | SQL Server-backed API-key store sharing **one row set** across clustered hosts (e.g. a central pair), instead of each node keeping its own SQLite file. DI wiring is `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called **instead of**`AddZbApiKeyAuth` (it calls that internally). Also ships `SqliteApiKeyStoreImporter` — a one-shot per-node SQLite → shared store copy a host runs once at boot. | `Abstractions`, `ApiKeys`, `Microsoft.Data.SqlClient` |
| `ZB.MOM.WW.Auth.AspNetCore` | ASP.NET Core wiring for the **LDAP** provider only: `AddZbLdapAuth` (binds + start-time-validates `LdapOptions`, registers `ILdapAuthService`), plus `ZbCookieDefaults.Apply` (hardened cookie helper the consumer calls itself) and `ZbClaimTypes` constants. It does **not** wire API keys or cookie middleware — API-key DI is `AddZbApiKeyAuth` in the `ApiKeys` package. | `Abstractions`, `Ldap`, `Microsoft.AspNetCore.App` |
`ApiKeys` is NOT used by OtOpcUa (that app authenticates human operators via LDAP + cookies only; machine-to-machine access is out of scope).
@@ -29,7 +30,73 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
## Versioning
All four packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.1.4**. A single version bump in `Directory.Build.props` bumps all four packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
All five packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.5.0**. A single version bump in `Directory.Build.props` bumps all five packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
### 0.5.0 — passwordless group lookup (`ILdapGroupLookup`, ScadaBridge#46)
Includes every unpublished 0.4.2–0.4.4 change below; **0.5.0 is the first published version carrying them** (scadaproj#51 live-verified them against the NAM AD before publishing).
- **New `Abstractions` contract:**`ILdapGroupLookup.GetUserGroupsAsync(string username, CancellationToken ct)` → `LdapGroupLookupResult(Status, Username, DisplayName, Groups)`, with `LdapGroupLookupStatus` = `Found` | `UserNotFound` | `UserDisabled` | `AmbiguousUser` | `DirectoryUnavailable` | `NotSupported` | `ProviderDisabled`. It re-reads a known user's groups **without their password** — e.g. to refresh an active session when directory membership changes.
- **`LdapAuthService` implements it** (`AddZbLdapAuth` registers `ILdapGroupLookup` as the *same* singleton, so sign-in and lookup share one sticky endpoint preference). It binds as the **service account**, searches `UserNameAttribute` under `SearchBase`, and returns the same group set, canonical username and display name `AuthenticateAsync` would — through the same group extraction, canonicalisation, endpoint walk (`FallbackServers`, system-side failures only), timeouts and TLS policy. The name is normalised as `AuthenticateAsync` normalises it in `ServiceAccount` mode (trimmed).
- **Outcomes:** connect/TLS, service-account bind or search failing on every endpoint → `DirectoryUnavailable` (a statement about the directory, never about the user). `UserDisabled` comes from AD's `userAccountControl` ACCOUNTDISABLE bit (requested automatically; directories without the attribute never report it). `Found` may carry **zero** groups — sign-in refuses such a user, the caller decides. `BindMode = User` (no service account) and a blank `ServiceAccountDn` answer `NotSupported` without touching the network; `Enabled = false` answers `ProviderDisabled`. Never throws (cancellation at entry only); credentials are never logged.
- A host that registers its own `ILdapAuthService` (a dev stub, a decorator) gets a lookup that answers `NotSupported`.
- **Consumer impact:** additive. No existing signature or enum value changed in 0.5.0 itself; the 0.4.2–0.4.4 notes below still apply to anyone coming from 0.4.1.
- **Live-verified 2026-11-12 against the NAM AD** (both DCs, LDAPS 636, StartTLS 389, the Global Catalog 3269 with a forest-wide base): lookup groups == sign-in groups (176), a real disabled account → `UserDisabled`, unknown → `UserNotFound`, failover to either DC, all-dead / wrong service password → `DirectoryUnavailable`, `User` mode → `NotSupported`. Plus GLAuth (redundant local pair) integration tests `GLAuthGroupLookupTests`.
### 0.4.4 — user bind proves the bound account is the one whose groups it returns (first published in 0.5.0, scadaproj#40)
- **Security fix (Auth-007), `BindMode = User` only.** The bind name (`UserBindTemplate`) and the group search (`UserNameAttribute`) are two separate directory lookups and could name two accounts — e.g. AD resolves `jsmith@corp.com` to the account with that explicit UPN while `(sAMAccountName=jsmith)` finds a different one, so a user could sign in with another account's groups. The service now only accepts the entry it found when it is provably the account that bound:
- it asks the directory with RFC 4532 **Who-am-I** after the bind; a `dn:<dn>` answer must equal the entry's DN (`distinguishedNameMatch`), and AD's `u:DOMAIN\name` answer must equal the entry's `sAMAccountName` (requested automatically);
- a server without Who-am-I (GLAuth answers protocol error 2) is supported only with a **DN template** (contains `=`), whose bind DN must equal the entry's DN;
- anything else — a mismatch, an unanswered Who-am-I with a UPN or down-level template — is `GroupLookupFailed`, logged at Warning, and never retried on a fallback endpoint.
- **`{0}` is now RFC 4514-escaped in a DN template**, so a typed name such as `x,ou=Admins` cannot rewrite the bind DN.
- **Behaviour change for consumers:** AD with a down-level or UPN template is unaffected for accounts whose bound identity and searched entry agree (AD has supported Who-am-I since Windows Server 2003). A UPN template now refuses accounts whose UPN prefix differs from `sAMAccountName` instead of risking the wrong account. A **non-AD directory** in `User` mode needs either Who-am-I support or a DN template that yields the entry's DN *exactly as the search returns it* — for GLAuth that is `cn={0},ou=<group>,ou=users,<base>`, not `cn={0},ou=<group>,<base>`. ServiceAccount mode is unchanged.
- The validator's `UserBindTemplate` message now leads with `EXAMPLE\{0}`. No `Abstractions` contract change (XML docs only).
- **AD forests (scadaproj#50).**`sAMAccountName` is unique per *domain*, so with a search base that spans a forest (a Global Catalog) a matching `sAMAccountName` alone could be another domain's account of the same name. For AD's `u:DOMAIN\name` answer the entry must now also be proven to be that domain's account:
- where the directory computes `msDS-PrincipalName` for the entry (a base read; AD's `DOMAIN\name`, derived from `objectSid`), it must equal the Who-am-I answer, domain included;
- otherwise the name must have exactly one holder under `SearchBase`, and it must be the entry found — no extra search is made when `UserNameAttribute` is `sAMAccountName` and the user typed the bound name, because the user search already proved that. Two holders → `GroupLookupFailed`.
- A single-domain AD (BTDB Manager) is unaffected. **Live-verified against the NAM AD 2026-11-12 (scadaproj#51):**`User` mode with `NAM\{0}` and the UPN template both sign in with the same groups as `ServiceAccount` mode; AD answers Who-am-I as `u:NAM\<sAMAccountName>` and serves `msDS-PrincipalName` on a base read (DC and Global Catalog); a forest-wide Global Catalog base (`DC=zimmer,DC=com`) signs in correctly; and mutating the `msDS-PrincipalName` comparison makes every user-bind sign-in fail `GroupLookupFailed`, so that check is what decides against real AD.
- **`ApiKeyFailure.StoreUnavailable` (Auth-010, `Abstractions` addition).** `ApiKeyVerifier.VerifyAsync` no longer lets a key-store fault escape: a `FindByKeyIdAsync` that throws (shared SQL Server unreachable, a SQLite I/O fault) returns `Succeeded = false` with the new `StoreUnavailable` failure, so it takes the host's normal — audited — auth-failure path instead of an unhandled 500 / gRPC `Unknown`. Cancellation still throws. The value is appended, so existing numeric values are unchanged; a host may answer it with 503 rather than 401. A misconfigured `SqlServerApiKeyOptions` still throws when the store is first resolved (outside `VerifyAsync`).
- **SQL Server store: no truncation, exact key ids (Auth-008).** The `ApiKeys.SqlServer` stores now behave like the SQLite store they port:
- `CreateAsync` throws `ArgumentException` for a key id over 64, prefix over 16 or display name over 256 characters (it used to store them silently truncated), and every lookup of an over-64-character key id answers "not found" without querying (a truncated parameter could match a different key sharing its first 64 characters).
- Key-id matching is **exact** — case- and trailing-space-sensitive — via a binary comparison added to every `key_id` predicate; the primary-key seek is unchanged and there is **no schema change**. A token presented with a case-altered key id (`LINE1` for `line1`) now fails `KeyNotFound` instead of resolving.
- `SqliteApiKeyStoreImporter` reports instead of hiding: `ApiKeyImportResult.KeyIdCollisions` lists source keys whose id differs only in case from one already in SQL Server (the primary key follows the database collation and cannot hold both — re-mint those keys), and `KeysRejected` lists over-length keys; neither is counted in `KeysSkipped` any more. Both are new `init` properties, so existing code is unaffected.
- Audit text (`key_id`, `event_type`, `remote_address`) is still truncated to 64 characters — deliberately, and now documented: an audit append runs after the action it records and must not fail it.
- **Canonical username from a multi-valued attribute (Auth-009).**`LdapAuthResult.Username` is now the `UserNameAttribute` value that equals the typed name ignoring case — `cn: [John Smith, jsmith]` signs `jsmith` in as `jsmith`, not whichever value the server lists first. The first value is used only when none matches. Both bind modes.
### 0.4.1 — LDAP review fixes
- **Canonical username (both modes):** the search now also requests `UserNameAttribute`, and `LdapAuthResult.Username` is the entry's first value of it (the directory's spelling), so `JDoe` and `jdoe` sign in as one identity in consumers' audit columns. Falls back to the normalised typed name when the entry has no such value.
- **Behaviour change for `ServiceAccount`-mode consumers:** an empty (or null) password is now refused by the library as `BadCredentials` before any network call, in both bind modes. Previously only `User` mode refused it; in `ServiceAccount` mode the re-bind as the user's DN with an empty password is an unauthenticated bind that AD reports as success, so a consumer without its own edge check admitted anyone.
- **Template guidance:** prefer the down-level `UserBindTemplate``EXAMPLE\{0}` — it maps exactly to the `sAMAccountName` the group search uses. A UPN template (`{0}@example.com`) is only correct where every account's UPN prefix equals its `sAMAccountName`; AD resolves UPN binds against the explicit `userPrincipalName`, so a mismatch can fail sign-in or bind a different account than the one whose groups are read.
- No `Abstractions` contract change.
### 0.4.0 — LDAP sign-in by user bind (opt-in)
- **`LdapOptions.BindMode`** (`ServiceAccount` default | `User`), **`UserBindTemplate`** (`{0}` = the login name, e.g. `EXAMPLE\{0}` — preferred, see 0.4.1 — or `{0}@example.com`) and **`AcceptDomainQualifiedNames`** (default `true`).
- In `User` mode `LdapAuthService` binds as the user; success authenticates them and the same connection reads their entry (`UserNameAttribute`, `DisplayNameAttribute`, `GroupAttribute`). No `ServiceAccountDn` / `ServiceAccountPassword` is needed or validated.
- Failure mapping: empty password or an unusable name → `BadCredentials` with no network call (AD treats an empty-password simple bind as an unauthenticated success). Bind result 49 → `BadCredentials`, final — never retried on a fallback DC, so one wrong password is one lockout strike. Any other bind error, a connect failure or a search failure → the system-side bucket (`ServiceAccountBindFailed`) and the next endpoint. Zero/several entries or no groups after a good bind → `GroupLookupFailed`.
- `DOMAIN\name` / `name@suffix` are reduced to `name` (or refused when `AcceptDomainQualifiedNames` is `false`), so the result's `Username` has one spelling (0.4.1 also canonicalises its case from the entry).
- No `Abstractions` contract change beyond the new enum and option properties; `ServiceAccount` mode is unchanged.
- `UserBindTemplate` output is used verbatim as the bind name, so use a down-level (`EXAMPLE\{0}`, preferred) or UPN (`{0}@example.com`, only where every UPN prefix equals the `sAMAccountName`) template; a DN template is not escaped.
### 0.3.0 — SQL Server API-key store (ScadaBridge central pair)
- **`IApiKeyStoreMigrator`** added to `Abstractions`: creates or upgrades an API-key store's schema; idempotent and safe to run concurrently from several nodes. `SqliteAuthStoreMigrator` implements it unchanged. `AddZbApiKeyAuth` registers the SQLite implementation via `TryAdd`, so a provider that registers its own migrator first (see below) wins — that ordering is the whole mechanism by which a different backend is swapped in.
- **New package `ZB.MOM.WW.Auth.ApiKeys.SqlServer`** + `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called instead of `AddZbApiKeyAuth` (it calls it internally after registering the SQL Server stores and migrator first).
- **`zbauth` schema v1** (three tables: `schema_version`, `api_keys`, `api_key_audit`; `DATETIMEOFFSET(7)` timestamps). `SqlServerApiKeyStoreMigrator` is concurrent-boot safe — a central pair's two nodes both migrate at every boot — with retry on SQL error numbers 1205/2714/1913/2627 and jittered backoff (measured: the flat backoff it replaced exhausted its retries roughly 1 in 60 eight-way boots). DBA-facing DDL: `docs/sql/zbauth-schema.sql`.
- **`SqliteApiKeyStoreImporter`** — idempotent insert-where-not-exists copy of an existing node's per-node SQLite keys and audit rows into the shared store; a row already present in SQL Server is never modified, so running it from both nodes of a pair converges on the union of their files.
- **`ApiKeyAdminCommands`** gained a ctor taking `IApiKeyStoreMigrator`; the existing ctor taking the concrete `SqliteAuthStoreMigrator` is retained (MxAccessGateway, HistorianGateway and ScadaBridge all construct it that way).
- SQLite remains the default store; **existing consumers need no code change to move to 0.3.0**.
- Live tests for the SQL Server package are gated on `AUTH_SQLSERVER_CONNSTR` and skip (not fail) when it is unset.
### 0.2.1 — LDAP AD continuation-referral fix
- `NovellLdapConnection.Search` now skips Active Directory continuation referrals (`LdapReferralException` from `Next()`) instead of letting one fail the whole search — a `SearchBase` at a domain root made every login fail as `ServiceAccountBindFailed` against a healthy directory. Non-referral faults still propagate untouched.
### 0.2.0 — version bump only
- No functional change; bumped the lockstep family version ahead of the 0.2.1 LDAP fix.
### 0.1.4 — API-key expiry (archreview G-2)
@@ -63,6 +130,8 @@ The GLAuth integration test (`GLAuthIntegrationTests`) is **skipped by default**
The 0.5.0 group-lookup tests (`GLAuthGroupLookupTests`) use the same variables plus an optional `ZB_LDAP_FALLBACK=host:port` for a second live directory (e.g. the redundant pair in `infra/glauth-redundant`, `:3893` + `:3894`, service account `cn=serviceaccount,dc=zb,dc=local`).
All other variables (`ZB_LDAP_SERVER`, `ZB_LDAP_PORT`, `ZB_LDAP_BASE`, `ZB_LDAP_USERATTR`) default to sensible GLAuth values and are optional. The test also probes TCP reachability before attempting auth and skips if the server is not contactable.
/// The failure reason is discriminated for the caller/audit pipeline, but the verifier returns a
/// structured result rather than throwing (the caller decides the opaque client-facing message).
/// The only exception path is cancellation. A successful identity carries the key's scopes and the
/// The only exception path is cancellation: a key-store lookup that faults is reported as
/// <see cref="ApiKeyFailure.StoreUnavailable"/> (since 0.4.4), and a failed last-used write is ignored. A successful identity carries the key's scopes and the
/// opaque <c>ConstraintsJson</c> blob (which the verifier does not interpret); it never carries the
/// presented secret, the pepper, or the stored secret hash.
/// </remarks>
@@ -44,7 +45,24 @@ public sealed class ApiKeyVerifier(
}
// 3. Look up the record (including revoked ones) so we can discriminate not-found vs revoked.
@@ -4,7 +4,7 @@ Startup configuration-validation library for the **ZB.MOM.WW SCADA family** (OtO
The library normalizes the three-project configuration-validation surface: a failure-accumulating `IValidateOptions` base, reusable rule primitives, a bind+validate+`ValidateOnStart` DI extension, and a pre-host `ConfigPreflight` aggregator for raw `IConfiguration` — so the plumbing is written once and domain rules stay per-project.
**Built at 0.1.0. Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes). Adoption tracked in `~/Desktop/scadaproj/components/configuration/GAPS.md`.
**Built at 0.1.0. Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01, and by HistorianGateway since;** all four pin `0.1.0` (the only feed version) on their pushed origin default branches (verified 2026-11-09). Adoption tracked in `~/Desktop/scadaproj/components/configuration/GAPS.md`.
---
@@ -66,7 +66,7 @@ ZB.MOM.WW.Configuration/
## Status
Part of the **scadaproj component-normalization family** — this is the configuration + validation component. Built at **0.1.0**. **Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (local default branches; not yet pushed to remotes) — per-app result is tracked in:
Part of the **scadaproj component-normalization family** — this is the configuration + validation component. Built at **0.1.0**. **Adopted by OtOpcUa, MxAccessGateway, and ScadaBridge on 2026-06-01** (since pushed; HistorianGateway also pins `0.1.0` — all four verified on their origin default branches 2026-11-09) — per-app result is tracked in:
- `~/Desktop/scadaproj/components/configuration/shared-contract/ZB.MOM.WW.Configuration.md` — proposed shared-library API
- `~/Desktop/scadaproj/components/configuration/current-state/` — per-project current state (code-verified)
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
### Narrative
The configuration component is fully populated: a normalized [`spec`](components/configuration/spec/SPEC.md), a
[`shared-contract`](components/configuration/shared-contract/ZB.MOM.WW.Configuration.md), three
[`current-state`](components/configuration/current-state/) docs, and an adoption [`GAPS`](components/configuration/GAPS.md)
backlog. Shared = the `IValidateOptions<T>` failure-accumulation base (`OptionsValidatorBase<T>`) +
**Adopted across all three apps on 2026-06-01** (OtOpcUa, MxAccessGateway, ScadaBridge) on each repo's
local default branch (`main`/`master`) — merged, and since pushed to origin (the "not yet pushed" caveat this
record carried is stale; all four apps, HistorianGateway included, pin `0.1.0` on their origin default branches as of 2026-11-09); the package was first
published to the Gitea feed. Behaviour-preserving onto `OptionsValidatorBase`/`AddValidatedOptions`
for MxGateway + ScadaBridge (validator messages byte-identical), `StartupValidator` → `ConfigPreflight`
for ScadaBridge, and net-new `Ldap`/`OpcUa` validators for OtOpcUa. Per-app result tracked in
<Description>Startup configuration-validation toolkit for the ZB.MOM.WW SCADA family: a failure-accumulating IValidateOptions base, reusable rule primitives (port, host:port, required, positive-duration, one-of, min-count), a bind+validate+ValidateOnStart DI helper, and a pre-host ConfigPreflight aggregator for raw IConfiguration. Extracts the validation plumbing the apps share; domain rules stay per-project.</Description>
Shared Galaxy object-hierarchy SQL browse + `galaxy_repository.v1` gRPC service for the **ZB.MOM.WW SCADA family** (consumed by HistorianGateway and mxaccessgw). Build/test from this directory: `dotnet test`.
## 0.2.2 (2026-11-09, UNPUBLISHED)
Version bumped in `Directory.Build.props`, not packed or pushed. P3 review fixes (scadaproj#47):
- 003: parent cycles are broken at the lowest gobject id so their members stay browsable.
- 004: Stale threshold is `max(5 min, 3 x DashboardRefreshIntervalSeconds)` (new optional `IOptions` ctor
parameter on `GalaxyHierarchyCache`); the projected-Stale entry is memoized per source entry.
- 005: `WatchDeployEvents` subscribers never see a non-increasing `Sequence`.
- 006: container-building tests for `AddZbGalaxyRepository`.
- 007: `System.Security.Cryptography.Pkcs` pinned at 9.0.21 (clears NU1902/NU1903 for consumers).
Version bumped in `Directory.Build.props`, not packed or pushed; consumers stay on `0.2.0`.
- GalaxyRepository-001 (scadaproj#41): the per-cache-entry filter memos of both projectors are now
`ProjectionMemo<T>`, capped at 64 signatures (FIFO; an evicted signature is recomputed, same answer).
- GalaxyRepository-002 (scadaproj#42): a browse-subtree scope (`Plant/LineA/*`) no longer hides the
allowed subtree's ancestors. `BrowseChildren` and `DiscoverHierarchy` return them as
**navigation-only** nodes — attributes stripped, only the path to an allowed object listed, siblings
still hidden. **Behaviour change for consumers:** a scoped `DiscoverHierarchy` now includes those
ancestors (`GalaxyHierarchyQueryResult.NavigationAncestorCount` says how many); `WatchDeployEvents`
still counts allowed objects only. Semantics documented on `IGalaxyBrowseScopeProvider`.
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
### Narrative
The Galaxy Repository component normalizes the **Galaxy object-hierarchy SQL browse + reusable gRPC service**
that was previously embedded in `mxaccessgw`. Shared = canonical `galaxy_repository.v1` proto (wire-compatible
with `mxaccessgw`'s existing contract so OtOpcUa's `GalaxyRepositoryClient` is unaffected), the SQL browse
**Consumed by HistorianGateway** as a `PackageReference` from that Gitea feed, pinned at `0.2.0` (originally a
cross-repo `ProjectReference` to this scadaproj tree; switched to the feed package 2026-06-24; bumped `0.1.0` → `0.2.0` 2026-06-25).
**mxaccessgw has adopted it** (its Server calls `AddZbGalaxyRepository` / `MapZbGalaxyRepository`; the
`galaxy_repository.v1` wire contract is identical, so OtOpcUa and ScadaBridge clients are unaffected). Build/test from `ZB.MOM.WW.GalaxyRepository/`: `dotnet test`.
<Description>Read-only Galaxy object-hierarchy browse library for the ZB.MOM.WW SCADA family. Provides a SQL provider for the Galaxy Repository database and a canonical gRPC service for exposing the hierarchy to modern .NET 10 clients — extracted from MxAccessGateway so any consumer can browse the Galaxy without loading 32-bit COM.</Description>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.