From ede5275d3ae52e3519dc45a696906b1ebeb60003 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 21 Jul 2026 03:30:36 -0400 Subject: [PATCH] docs(index): OtOpcUa address-space availability guarantee; track the OtOpcUa LocalDb design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #485/#486 fixes changed nothing structural about OtOpcUa — same drivers, same dual-namespace scheme, same Galaxy flow — so the row is mostly untouched. What DID change is a fact any OPC UA consumer depends on, and ScadaBridge is mid-cutover onto exactly this address space: a transient ConfigDb error used to empty the served address space (a PureRemove of every node, observed live and it stayed empty), stop every driver and clear every subscription, while still reporting the deploy Applied. A consumer would have seen all bindings go bad with nothing upstream admitting a problem. Recorded on the OtOpcUa row. Also fixes real drift found while checking: the OtOpcUa LocalDb adoption design was sitting UNTRACKED here and unreferenced by the index, even though the LocalDb row claims the adoption designs live in this repo. Now committed, referenced alongside the ScadaBridge one, and its stale header corrected — it still said Phase 1 was "not yet executed" against lib 0.1.1, when Phase 1 shipped and merged against 0.1.3. Phase 2 designed but not executed, on both the doc and the row. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 4 +- .../2026-07-20-otopcua-localdb-design.md | 204 ++++++++++++++++++ 2 files changed, 206 insertions(+), 2 deletions(-) create mode 100644 docs/plans/2026-07-20-otopcua-localdb-design.md diff --git a/CLAUDE.md b/CLAUDE.md index 39476c8..94843dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-th | Project | Location | Stack | Repo | Summary | |---|---|---|---|---| -| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient) **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings, coordination pending):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. | +| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient) **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings, coordination pending):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** | | **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. | | **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib (consumed as a Gitea-feed package). No COM, no x86 worker. **Dev:** two plaintext endpoints from `appsettings.Development.json` — dashboard on `:5220` (HTTP/1.1), gRPC h2c on `:5221`. **Production:** single `Kestrel:Endpoints:Https` endpoint with `Protocols: Http1AndHttp2` multiplexes dashboard + gRPC over one TLS port (ALPN); warn-only if no TLS endpoint configured (valid behind a reverse proxy / Kubernetes ingress; the warn predicate covers any non-Development environment, i.e. Production + Staging). In a non-Development environment the gateway also logs warn-only **production-readiness** checks (pending.md D2/D3) — relative runtime-artifact paths + secret hygiene (`ApiKeys:Mode=Disabled`, empty/dev-placeholder pepper, dev-placeholder LDAP password). **Owns `AVEVA.Historian.Client`** — imported into the repo at `histsdk/` (subtree, folded in with history 2026-06-28; the old "vendored / re-vendor from upstream" model is **retired** — `histsdk/**` is ordinary owned source, edited in place; the gateway builds it directly). Store-forward uses a crash-safe FasterLog append-only outbox (`Microsoft.FASTER.Core` 2.6.5; `CommitMode` PerEntry/Periodic), not SQLite. **Handshake amortization (pending.md A1) done + live-validated** — a default-on leased-session pool (`Historian:SessionPool`) reuses pre-authenticated sessions across reads/writes/status ops/tag-browse/metadata (~4.7× measured; probe and blocks stay per-call), with a `<~15 s` keepalive + reactive re-auth, surfaced via a `PooledHistorianClient` facade so services are unchanged; the `HistorianSession` primitive lives in the owned `AVEVA.Historian.Client` under `histsdk/`; browse/metadata + SendEvent broadening merged to `main` + pushed. **`SendEvent` is also amortized** via a **separate, parallel event-session pool** (`Historian:EventSessionPool`, default-on; v8/ECDH auth — kept distinct from the v6 pool), warranted by a GREEN v8 Event-session reuse spike (~10–16×); `ReadEvents` stays per-call / gated (C2). The full offline suite is green on macOS (0 warnings); the env-gated live historian + Galaxy integration suite exercises the amortized path and otherwise skips without a live server. **Part C minimal (pending.md C1/C3a) merged to `main`:** `Int8`/`UInt8` live write types un-gated + live-proven against `wonder-sql-vd03` (in the owned `histsdk/` SDK); the 2023 R2 gRPC interface-version integers recorded as evidence (C3a); `UInt1` attempted but **server-blocked** (the historian accepts `EnsureTags(UInt1)` yet stores a degenerate analog tag) → re-gated fail-closed. **C2 closed won't-fix (2026-06-26):** event reads are server-gated on the 2023 R2 historian over **both** transports — gRPC retrieval-server-gated (0 rows scoped to a managed connection), and the WCF certificate transport + auth DO reach the historian cross-platform (CM_EVENT registers on the `0x501` event connection) but the query still returns 0 rows: the same server-side per-connection row gate. (An earlier note saying "WCF not served on 2023 R2" was a test error vs the reverse tunnel.) Not client-fixable. Reusable SDK wins: `ConnectViaAddress` (WCF Via for tunneled access), `EventReadConnectionModeOverride`. **SQL-path `ReadEvents` (merged 2026-06-26) — the practical workaround for the C2 gate:** event reads now **ship** via the historian's `Runtime.dbo.Events` SQL view (config-gated `RuntimeDb:EventReadsEnabled` + `EventReadMaxRows`), mirroring the SQL live-write path (no COM/native); live-proven streaming real events (incl. an INSQL NOT-NULL `CAST` fix the live test caught) while the native event-query stays gated — so **event reads work** despite the native dead-end. **All of Part C + C2 + SQL-ReadEvents are merged to `origin/main`** (gateway @ `fabab1a`, histsdk @ `f0a1b04`). **Follow-ups merged to `main` (2026-06-27):** `historiangw` **PR #6** — `WriteLiveValues` UTC→server-local timestamp fix (live-validated exact) + a documented `SendEvent` `Source_Object` protocol limitation (`pending.md` C4); the Plan-1 .NET **client lib + packable Contracts** (PR #5); and a C2 cross-ref docs PR (#4). **First consumer (merged):** OtOpcUa **PR #423** (→ `master`) adopts `ZB.MOM.WW.HistorianGateway.Client` as its sole historian read/write backend, retiring its bespoke Wonderware historian driver. **Local Docker (2026-06-29):** `HistorianGateway/docker/` runs the sidecar in a container against the REAL wonder datasources (historian `wonder-sql-vd03:32565` TLS + Galaxy SQL `wonder-app-vd03`/`ZB`), dashboard login + gRPC API-key auth disabled — verified `/health/ready` Healthy. Runtime-only image: publish framework-dependent on the host (authed Gitea feed → no Docker restore), COPY into `aspnet:10.0`; secrets via gitignored `env_file`; isolated compose project `zb-historiangw`; container egress needs the host VPN up. **Client surface (`clients/dotnet/`):** the published `ZB.MOM.WW.HistorianGateway.Client` lib + a `ZB.MOM.WW.HistorianGateway.Client.Cli` smoke CLI (`probe`/`read-raw`/`browse`, JSON out; in the `.slnx`, not packed). **Current: `origin/main` @ `56a70df`** — **AHC-002 string READS landed via native RE from the Mac (PR #17 merged 2026-07-16):** the SDK now parses string historian result-buffer rows into `HistorianSample.StringValue` (a per-row `valueType` discriminator — `1`=numeric→`double`, `3`=string; value = `u32 charLen` + UTF-16LE), and the gateway forwards `string_value` through `HistorianRead.ReadRaw` — **no contract change, no package bump** (the proto already carried the field; mapper already forwarded it); live-proven against 2023 R2. String **tag-create** is now doable via SQL `aaStringTagInsert` (`SqlStringTagProvisioner`, test tooling — single + double byte). String **historical writes** were probed (bounded 6-candidate native blob probe) but **non-convergent** — tag-info resolves a **real GUID + SingleByteString** (unlike the Int1/UInt1 degenerate stub) so writes aren't blocked at resolution, but `BSuccess` is **not an oracle** (all framings optimistically accepted) and none round-tripped → **deferred, needs a native-client write wire capture** (blind guessing can't finish). Eight follow-up issues filed (`historiangw` #18–#25): datatype gaps (native string tag-create #18, string writes #19, DateTime/FileTime #20, Guid/Structure/Event #21, Int1/UInt1 discrete-create #22 [follow-on to #11], DoubleByteString read-verify #23) + **SQL-fallback trackers** (event reads via `Runtime.dbo.Events` #24, live writes via `History`/`v_StringHistory` #25). Also folded a doc-truth reconcile of the "Key structural decisions" bullets (GalaxyRepository `PackageReference`, Contracts/Client 0.3.0 published, OtOpcUa on 0.3.0). Prior: **ArchReview #2 (cycle-2) remediation MERGED 2026-07-14 (PR #12):** all 8 workstreams A–H — store-forward drain honesty (`IsRetryableDeliveryFault`, typed `HistorianOutboxFullException`→`ResourceExhausted`), streaming-lease caps (per-key cap 8→4, `MaxStreamLeaseHold`, bounded `EndQuery` teardown, client `StreamTimeout` default null→10min), a **live idle-out capture (GREEN-A, native `(type 4, code 51)`) that narrowed the session-expiry classifier and closed the stability-F5 tail**, histsdk residue (typed event-send `FromNativeError`, `HistorianProtocolIntegrityException`, IP/host scrub), read-path caps (`MaxAggregateIntervals`/`MaxAtTimeTimestamps`, 5M→`ResourceExhausted`, verify-once-per-connection version gate, batched audit drain), release hygiene (csproj `` sole authority + packed-README/AnyCPU/FQDN/CHANGELOG guards), security/ops (opt-in `ForwardedHeaders` trust list, k8s key-bootstrap runbook, dashboard pool/dead-letter signals), and doc-truth #2 (`Historian:VerifyServerInterfaceVersion` now a real fail-closed key). Verified 0-warn, offline 1388 pass + live suite green vs `wonder-sql-vd03`. **Contracts+Client 0.3.0 PUBLISHED** to the `dohertj2-gitea` feed (2026-07-14; carries the `StreamTimeout` default change) and **OtOpcUa bumped to 0.3.0** (`lmxopcua master @ 7f79cd59`, PR #440 — only consumer delta is the StreamTimeout default; driver tests 103/0). **Boolean historization remapped Int1→Int2 (2026-07-14, `lmxopcua master @ 38e21df2`, PR #442, closes #441):** the R2-06 live gate surfaced that the historian's `Int1` analog-tag creation path is server-degenerate (`EnsureTags(Int1)` stores an unusable stub) — filed gateway-side as `historiangw` #11, documented there as an explicit evidence-based exclusion (PR #16), and routed back to OtOpcUa as issue #441. OtOpcUa's `HistorianTypeMapper` now maps `Boolean → Int2` (0/1 integer); the value-write path already sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips cleanly and a pre-existing Float Boolean tag retypes cleanly (both are supported analog types). Closes the last R2-06 red test → live gate **6/6**; gateway driver unit 103/103. **archreview round-2 deferred gates closed 2026-07-14** (`lmxopcua master @ 30672888`): **R2-04 T13+T15 LIVE-PASSED** (PR #444) — the behavior-affecting S4 primary-gate default-deny verified on a 2-node docker-dev rig (ServiceLevel 250/240; boot-window write rejected `not primary (role unknown)` + denial meter `reason=role-unknown`; steady-state secondary rejected + node reverted; primary write reaches device) plus the in-process `PrimaryGateFailoverTests` (delivered-snapshot drives the gate); and the pre-existing red S7.Cli phrase-scan test fixed (PR #443, restored a stripped rationale comment). Round-2 remainder is now only infra-gated: R2-08 live-GLAuth outage, R2-01 S7 SYN-blackhole. **The full `*.IntegrationTests` sweep is now COMPLETE (2026-07-15, `lmxopcua master @ 152a5645`):** ran all 10 driver + server integration suites serially (on the Mac against native fixtures on the `10.100.0.35` Docker host), then closed all 7 resulting follow-ups (PRs #446–#452) — **every driver + server integration suite is verified GREEN with ZERO OtOpcUa production regressions;** every original "failure" was fixture/harness/test rot, not product. Highlights of cross-repo relevance: (a) **the Host integration harness's LDAP swapped from the retired `bitnami/openldap:2.6` to GLAuth** (PR #451) — the harness now runs its own ephemeral GLAuth container on `:3894` (seeded by `tests/.../Host.IntegrationTests/glauth/config.toml`, distinct from the shared dev GLAuth on `:3893`), unifying the whole family's opt-in real-LDAP path on GLAuth and removing the lone OpenLDAP outlier; (b) the deploy-E2E "amd64-emulated-SQL timing" complaint was **misdiagnosed** — those tests had only ever run against the EF in-memory provider (no FK enforcement) and lacked the `ServerCluster`/`ClusterNode` seed the real-SQL `NodeDeploymentState` FK requires; fixed with a harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST`/`_LDAP_HOST` overrides so the fixtures run on the native-amd64 Docker host (PR #452). See `archreview/plans/INTEGRATION-SWEEP-STATUS.md` for the full record. Cycle-2 gated tail: CI first-green-run (needs a Gitea act_runner), the histsdk protobuf/Grpc.Tools major bump (03-CC-8, blocked on migrating off obsolete `GrpcWebHandler.HttpVersion`), and the 1.0.0 TLS-default flip. The prior cycle-1 **ArchReview remediation is fully merged** (Now + Next tiers 2026-07-09 @ `508b9c3`; **Later tier 2026-07-13** via a coordinated two-PR merge, `12385c8` + `6a24a30`). Off a 2026-07-08 six-report architecture review (`archreview/`) turned into 8 workstream plans (`docs/plans/2026-07-09-archreview-*`; tracker `...-remediation-tracking.md`), shipped clean under `TreatWarningsAsErrors`: a hard-fail `ResilienceComposition` validator (StoreForward+Redundancy both-on), corruption-tolerant outbox + dead-letter/overflow knobs, `/auth/login` rate-limit, server-side `max_values` cap, a headless `apikey` CLI, histsdk parser hardening, Gitea Actions CI + warnings-as-errors on all trees, pool observability (lease-wait/waiter/counters) + session-pool health probe, and a typed histsdk fault taxonomy with type-based classification + session churn-guard. **Contracts+Client 0.2.0 published** to the `dohertj2-gitea` feed (2026-07-09; Int1 proto3-optional, opc_quality doc, CA-chain pin). **Cross-repo 0.2.0/G-2 chain COMPLETE:** OtOpcUa bumped to Client+Contracts `0.2.0` (2026-07-10, `master @ f6eaa267`; M3 `opc_quality` gate clear), and the shared `ZB.MOM.WW.Auth.ApiKeys` gained `ExpiresUtc` at `0.1.4` — consumed by HistorianGateway (`apikey create --expires` + dashboard Expires column) and mxaccessgw (verifier enforcement). **Later tier (roadmap 16–21) MERGED to `main` 2026-07-13** (coordinated two-PR merge; both branches now deleted): **PR #8** (`archreview-later-21-and-maxsessions`, merge `12385c8`) = Later-21 (audit off the mutating-RPC critical path + `WriteLiveValues` SQL batching + composed-failure/shutdown tests) + the **MaxSessions 4→8 raise** (live concurrent-session spike GREEN); **PR #9** (`later-16-page-through-streaming`, merge `6a24a30`) = Later-16 **ReadRaw page-through streaming** (streams per server result buffer; pool holds the lease across the enumeration; new `Historian:ReadLimits:AllowStreamingBeyondCeiling`), Later-17 **redundancy honest-hardening** (exposes the `Redundancy:WriteAcknowledgement`/`WriteFanout`/`FailureThreshold` ack policy the docs already claimed), Later-18 **per-key concurrency cap** (`ApiKeys:MaxConcurrentCallsPerKey` → gRPC `ResourceExhausted`) + api-key-rotation runbook, Later-19 CHANGELOG-discipline guard test, Later-20 **K8s manifests + hardened prod Dockerfile + operator runbooks** (`deploy/k8s/`, `docker/Dockerfile.production`, `docs/runbooks/`). The two PRs conflicted on `CLAUDE.md`/tracking/`ValidatorTests`, resolved as the union (MaxSessions lands at **8**). Live-proven vs `wonder-sql-vd03`; offline gateway 729 pass / client 140 pass / 0 warnings. **Only remaining archreview item: the secure-by-default TLS-default flip, deferred to 1.0.0** (pre-announced in the client 0.2.0 CHANGELOG). (Prior tip `93c6051` = the 2026-07-07 CommentChecker XML-doc sweep.) | @@ -149,7 +149,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; production enablement would need real KEK distribution (own decision/issue). Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; only mxgw is Bootstrap-free) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `