From ff073450dc6b3e45f8453cbf74cec4f4aef94b74 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 30 Jul 2026 03:42:45 -0400 Subject: [PATCH] =?UTF-8?q?docs(index):=20OtOpcUa=20=E2=80=94=20tag-set=20?= =?UTF-8?q?drift=20detection=20REMOVED=20(#523,=20merge=202a0461a1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry claimed the periodic tag-set drift check had shipped. It was taken back out two days later and is not in master. Corrected in place rather than deleted, so the record shows both what landed and what became of it. Carries forward the consequence a reader needs: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so they now have no tag-change signal at all. Galaxy/TwinCAT/MTConnect/Mqtt raise natively and are unaffected. Still not consumer-facing to ScadaBridge -- no OPC UA wire-contract change either way, and the IRediscoverable operator prompt is untouched. Also records the rest of that merge: 13 stale skipped tests retired, a reflection guard over the deploy-time TagConfig gate, three silently-failing OpcUaClient integration tests fixed, and the two missing driver guides written. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index dcce607..32865ae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-th | Project | Location | Stack | Repo | Summary | |---|---|---|---|---| -| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient), a **read-only `Sql` DB-poll driver** (polls SQL Server tables/views on an interval → OPC UA variables; `connectionStringRef` env resolution, schema-walk address picker, live-gated on docker-dev — merged to `master` 2026-07-24), a **read-only `Mqtt` broker-subscribe driver with Sparkplug B** (one MQTTnet-5 client per driver, TLS+auth, hand-rolled reconnect; **Plain** mode binds a tag to a concrete topic, **SparkplugB** mode decodes vendored Eclipse-Tahu protobuf under one `spBv1.0/{groupId}/#` subscription and binds by the `group/edgeNode/device?/metric` tuple — birth/alias/seq-gap/rebirth state machine, death→STALE, a birth-driven browse picker with a scoped Request-rebirth NCMD; **write-through is deferred, every MQTT node is read-only**; live-gated on docker-dev against a Mosquitto TLS+auth broker + a project-owned C# Sparkplug edge-node simulator on `10.100.0.35:8883` — **merged to `master` 2026-07-27, merge `c3a2b0f7`, pushed to origin**), a **read-only `MTConnect` Agent driver** (one hand-rolled `System.Xml.Linq` HTTP/XML client per driver against a vendor-neutral MTConnect Agent — `/probe` builds the device model, `/current` primes, and a `multipart/x-mixed-replace` `/sample` long-poll pump feeds `ISubscribable.OnDataChange` with ring-buffer sequence-gap re-baselining and a heartbeat watchdog; `UNAVAILABLE` → `BadNoCommunication`; the TrakHound MTConnect.NET dependency was evaluated and **dropped** — neither pinned package can parse an MTConnect document or frame the sample stream socket-free; browse comes free from the Wave-0 universal discovery browser, so the driver ships no browser project; **write-back is deferred, every MTConnect node is read-only**; live-gated on an isolated docker-dev stack against a real `mtconnect/agent:2.7.0.12` + SHDR-adapter fixture on `10.100.0.35:5000` — **merged to `master` 2026-07-27, PR #506, merge `90bdaa44`, pushed to origin**), **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=;s=…` form + doc sweep. The binding re-author (legacy `ns=2` collides with v3 `raw`), native-alarm dedup across the raw+uns fan-out, and UNS-bound→RawPath alarm routing are DEFERRED to a live v3 rig — ScadaBridge #14 stays OPEN. Scope: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is now CLOSED** — Phase 7 drill D4 (2026-07-24, `c50ebcf7`) ran the 1-vs-1 crash-the-oldest drill on the three-mesh rig and every survivor stayed Up without self-downing. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **PROGRAM COMPLETE — Phases 0a/0b/1/2/3/4/5/6/7 all done + merged to `master` and pushed (Phase 7 `c50ebcf7`, bootstrap guard `d1dac87f`, 2026-07-24; Phase 6 was `2839deb5`, Phase 5 `35552a96`).** **Phase 2 (merge `6eaa81ca`)** added a `ClusterClient` command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership). **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — merged to `master` + pushed 2026-07-24, tip `2839deb5`; live gate RAN and caught three fix-forward defects — a boot-crash `StackOverflow` on every node from resolving `IClusterRoleInfo` inside the `AddAkka` configurator while the `ActorSystem` was still building, plus rig LDAP-missing on sites and a site-pair split-brain — all fixed, `792f28ec`)** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. **Phase 7 (`c50ebcf7`, 2026-07-24) closed the program:** 6/6 failover drills PASSED across all three pair types (graceful manual failover, auto-down both directions, crash-the-oldest, cold-start-alone, rejoin-as-Secondary), closing the auto-down gate open since Phase 0a. The manual-failover button is **MAIN-only by construction** (it acts on the local node's `Cluster.State` and the AdminUI runs only on the central pair) — site pairs fail over via auto-down only. A follow-on **bootstrap guard** (`d1dac87f`, opt-in `Cluster:BootstrapGuard:Enabled`, **default OFF**) closes simultaneous-cold-start split-brain: the lexicographically lower `host:port` node is the preferred founder, the higher node TCP-probes its partner and joins if reachable / self-forms if not — the order is chosen **before** the single `JoinSeedNodes`, so it can never re-form mid-handshake. **Deferment register (2026-07-27, `e08855fb`): `OtOpcUa/deferment.md`** — a source-verified inventory of unfinished work. **Two entries are consumer-relevant to ScadaBridge:** (1) **node-level ACLs are authored, deployed and NOT ENFORCED** — the `IPermissionEvaluator`/`TriePermissionEvaluator` subsystem has zero production consumers and `OtOpcUaNodeManager` never references it, yet operators can author `NodeAcl` rows that ship in every artifact; actual enforcement is LDAP roles plus a realm-qualified **write** gate, so **reads are ungated and authored deny rules have no effect** (do not assume OtOpcUa filters an address space per-user). (2) **a driver-reported remote change is an operator PROMPT, never an address-space rebuild** (`feat/deferment-remediation`, 2026-07-27, Gitea #518) — `IRediscoverable` was inert fleet-wide and is now consumed: a Galaxy redeploy / TwinCAT symbol-version bump / MTConnect Agent restart / Sparkplug rebirth raises a **"re-browse" chip** on the AdminUI `/hosts` page (carried on `DriverHealthChanged.RediscoveryNeededUtc`/`.RediscoveryReason`). It is **advisory**: the served address space is deliberately NOT changed, because v3 authors raw tags through the `/raw` browse-commit flow — so **the served tree still only changes on a config redeploy**, which makes it more stable than the driver docs imply, not less. Gitea #507 (discovered-node injection) is **closed as superseded** and its code deleted — it read two structures that are structurally empty in v3, so it could never have injected anything. `IHostConnectivityProbe` remains unconsumed (Gitea #521, a keep-or-delete decision). **That register's remediation is MERGED to `master` and pushed** (2026-07-28, merge `914d70bf`) — all six ranked items plus two deferred follow-ups: ACL non-enforcement made explicit in docs + UI warnings (wire-up = Gitea #520), the rediscovery signal above, **#516 driver config edits no longer silently discarded** (a changed `DriverConfig` now stop+respawns the driver instead of an in-place delta that five drivers ignored while the deploy sealed green — **operationally visible: a config edit now costs a driver reconnect**), AdminUI dispatch-map parity guards, plan bookkeeping, and the driver **stability tiers documented as inert** (every driver runs Tier A; recycle machinery dormant — Gitea #522). Two follow-ups the first pass had deliberately left unbuilt also landed: Sql/FOCAS now re-parse config in place too (each derives its connection/backend from config, so the parse was made **atomic** — options + dialect + connection string, or options + backend, adopted together — rather than half-applying a change), and **tag-set drift detection** for the four genuinely-browsable drivers (FOCAS, TwinCAT, MTConnect, AbCip re-browse every 5 min and raise the same `/hosts` re-browse prompt when the device's tag set differs from what was authored; gated on `SupportsOnlineDiscovery` so the five drivers that merely echo authored config are never checked, since for them the comparison is a tautology). **Not consumer-facing to ScadaBridge** — no OPC UA wire-contract change; the drift signal is advisory and never rebuilds the served address space. Not yet live-gated. | +| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient), a **read-only `Sql` DB-poll driver** (polls SQL Server tables/views on an interval → OPC UA variables; `connectionStringRef` env resolution, schema-walk address picker, live-gated on docker-dev — merged to `master` 2026-07-24), a **read-only `Mqtt` broker-subscribe driver with Sparkplug B** (one MQTTnet-5 client per driver, TLS+auth, hand-rolled reconnect; **Plain** mode binds a tag to a concrete topic, **SparkplugB** mode decodes vendored Eclipse-Tahu protobuf under one `spBv1.0/{groupId}/#` subscription and binds by the `group/edgeNode/device?/metric` tuple — birth/alias/seq-gap/rebirth state machine, death→STALE, a birth-driven browse picker with a scoped Request-rebirth NCMD; **write-through is deferred, every MQTT node is read-only**; live-gated on docker-dev against a Mosquitto TLS+auth broker + a project-owned C# Sparkplug edge-node simulator on `10.100.0.35:8883` — **merged to `master` 2026-07-27, merge `c3a2b0f7`, pushed to origin**), a **read-only `MTConnect` Agent driver** (one hand-rolled `System.Xml.Linq` HTTP/XML client per driver against a vendor-neutral MTConnect Agent — `/probe` builds the device model, `/current` primes, and a `multipart/x-mixed-replace` `/sample` long-poll pump feeds `ISubscribable.OnDataChange` with ring-buffer sequence-gap re-baselining and a heartbeat watchdog; `UNAVAILABLE` → `BadNoCommunication`; the TrakHound MTConnect.NET dependency was evaluated and **dropped** — neither pinned package can parse an MTConnect document or frame the sample stream socket-free; browse comes free from the Wave-0 universal discovery browser, so the driver ships no browser project; **write-back is deferred, every MTConnect node is read-only**; live-gated on an isolated docker-dev stack against a real `mtconnect/agent:2.7.0.12` + SHDR-adapter fixture on `10.100.0.35:5000` — **merged to `master` 2026-07-27, PR #506, merge `90bdaa44`, pushed to origin**), **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=;s=…` form + doc sweep. The binding re-author (legacy `ns=2` collides with v3 `raw`), native-alarm dedup across the raw+uns fan-out, and UNS-bound→RawPath alarm routing are DEFERRED to a live v3 rig — ScadaBridge #14 stays OPEN. Scope: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is now CLOSED** — Phase 7 drill D4 (2026-07-24, `c50ebcf7`) ran the 1-vs-1 crash-the-oldest drill on the three-mesh rig and every survivor stayed Up without self-downing. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **PROGRAM COMPLETE — Phases 0a/0b/1/2/3/4/5/6/7 all done + merged to `master` and pushed (Phase 7 `c50ebcf7`, bootstrap guard `d1dac87f`, 2026-07-24; Phase 6 was `2839deb5`, Phase 5 `35552a96`).** **Phase 2 (merge `6eaa81ca`)** added a `ClusterClient` command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership). **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — merged to `master` + pushed 2026-07-24, tip `2839deb5`; live gate RAN and caught three fix-forward defects — a boot-crash `StackOverflow` on every node from resolving `IClusterRoleInfo` inside the `AddAkka` configurator while the `ActorSystem` was still building, plus rig LDAP-missing on sites and a site-pair split-brain — all fixed, `792f28ec`)** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. **Phase 7 (`c50ebcf7`, 2026-07-24) closed the program:** 6/6 failover drills PASSED across all three pair types (graceful manual failover, auto-down both directions, crash-the-oldest, cold-start-alone, rejoin-as-Secondary), closing the auto-down gate open since Phase 0a. The manual-failover button is **MAIN-only by construction** (it acts on the local node's `Cluster.State` and the AdminUI runs only on the central pair) — site pairs fail over via auto-down only. A follow-on **bootstrap guard** (`d1dac87f`, opt-in `Cluster:BootstrapGuard:Enabled`, **default OFF**) closes simultaneous-cold-start split-brain: the lexicographically lower `host:port` node is the preferred founder, the higher node TCP-probes its partner and joins if reachable / self-forms if not — the order is chosen **before** the single `JoinSeedNodes`, so it can never re-form mid-handshake. **Deferment register (2026-07-27, `e08855fb`): `OtOpcUa/deferment.md`** — a source-verified inventory of unfinished work. **Two entries are consumer-relevant to ScadaBridge:** (1) **node-level ACLs are authored, deployed and NOT ENFORCED** — the `IPermissionEvaluator`/`TriePermissionEvaluator` subsystem has zero production consumers and `OtOpcUaNodeManager` never references it, yet operators can author `NodeAcl` rows that ship in every artifact; actual enforcement is LDAP roles plus a realm-qualified **write** gate, so **reads are ungated and authored deny rules have no effect** (do not assume OtOpcUa filters an address space per-user). (2) **a driver-reported remote change is an operator PROMPT, never an address-space rebuild** (`feat/deferment-remediation`, 2026-07-27, Gitea #518) — `IRediscoverable` was inert fleet-wide and is now consumed: a Galaxy redeploy / TwinCAT symbol-version bump / MTConnect Agent restart / Sparkplug rebirth raises a **"re-browse" chip** on the AdminUI `/hosts` page (carried on `DriverHealthChanged.RediscoveryNeededUtc`/`.RediscoveryReason`). It is **advisory**: the served address space is deliberately NOT changed, because v3 authors raw tags through the `/raw` browse-commit flow — so **the served tree still only changes on a config redeploy**, which makes it more stable than the driver docs imply, not less. Gitea #507 (discovered-node injection) is **closed as superseded** and its code deleted — it read two structures that are structurally empty in v3, so it could never have injected anything. `IHostConnectivityProbe` remains unconsumed (Gitea #521, a keep-or-delete decision). **That register's remediation is MERGED to `master` and pushed** (2026-07-28, merge `914d70bf`) — all six ranked items plus two deferred follow-ups: ACL non-enforcement made explicit in docs + UI warnings (wire-up = Gitea #520), the rediscovery signal above, **#516 driver config edits no longer silently discarded** (a changed `DriverConfig` now stop+respawns the driver instead of an in-place delta that five drivers ignored while the deploy sealed green — **operationally visible: a config edit now costs a driver reconnect**), AdminUI dispatch-map parity guards, plan bookkeeping, and the driver **stability tiers documented as inert** (every driver runs Tier A; recycle machinery dormant — Gitea #522). Two follow-ups the first pass had deliberately left unbuilt also landed: Sql/FOCAS now re-parse config in place too (each derives its connection/backend from config, so the parse was made **atomic** — options + dialect + connection string, or options + backend, adopted together — rather than half-applying a change), and tag-set drift detection (FOCAS/TwinCAT/MTConnect/AbCip re-browsed every 5 min and raised the same `/hosts` prompt when the device's tag set differed from what was authored) — **which was REMOVED two days later and is NOT in `master`** (Gitea #523, merge `2a0461a1`, 2026-07-30): the check was recurring, unconfigurable (interval was a constructor parameter with no appsettings key) and never live-gated outbound traffic to real PLCs/CNCs/Agents, and how often to browse someone's plant equipment is an operational decision rather than an implementation detail. **Consequence to carry forward:** AbCip and FOCAS browse a live backend but implement no `IRediscoverable`, so they now have **no tag-change signal at all** — a PLC re-download is invisible until an operator re-browses by hand; Galaxy/TwinCAT/MTConnect/Mqtt raise natively and are unaffected. **Not consumer-facing to ScadaBridge either way** — no OPC UA wire-contract change, and the `IRediscoverable` prompt above is untouched. That same round (merge `2a0461a1`) also retired 13 stale skipped tests (6 revived onto the v3 raw/UNS shape, 3 folded into a widened parity corpus, 4 deleted as retired), added a reflection guard over the deploy-time TagConfig gate after finding MQTT shipped an `Inspect()` nobody called, fixed three OpcUaClient integration tests that had been silently failing on `master` (v3 made the driver resolve references through its authored `RawTags` table, so the tests' bare `ns=3;s=…` node ids failed locally at the resolver and never reached the wire), and wrote the two missing driver guides (`docs/drivers/Sql.md`, `docs/drivers/Calculation.md` — Sql had shipped with no user-facing doc at all). | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** | | **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. | | **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib (consumed as a Gitea-feed package). No COM, no x86 worker. **Dev:** two plaintext endpoints from `appsettings.Development.json` — dashboard on `:5220` (HTTP/1.1), gRPC h2c on `:5221`. **Production:** single `Kestrel:Endpoints:Https` endpoint with `Protocols: Http1AndHttp2` multiplexes dashboard + gRPC over one TLS port (ALPN); warn-only if no TLS endpoint configured (valid behind a reverse proxy / Kubernetes ingress; the warn predicate covers any non-Development environment, i.e. Production + Staging). In a non-Development environment the gateway also logs warn-only **production-readiness** checks (pending.md D2/D3) — relative runtime-artifact paths + secret hygiene (`ApiKeys:Mode=Disabled`, empty/dev-placeholder pepper, dev-placeholder LDAP password). **Owns `AVEVA.Historian.Client`** — imported into the repo at `histsdk/` (subtree, folded in with history 2026-06-28; the old "vendored / re-vendor from upstream" model is **retired** — `histsdk/**` is ordinary owned source, edited in place; the gateway builds it directly). Store-forward uses a crash-safe FasterLog append-only outbox (`Microsoft.FASTER.Core` 2.6.5; `CommitMode` PerEntry/Periodic), not SQLite. **Handshake amortization (pending.md A1) done + live-validated** — a default-on leased-session pool (`Historian:SessionPool`) reuses pre-authenticated sessions across reads/writes/status ops/tag-browse/metadata (~4.7× measured; probe and blocks stay per-call), with a `<~15 s` keepalive + reactive re-auth, surfaced via a `PooledHistorianClient` facade so services are unchanged; the `HistorianSession` primitive lives in the owned `AVEVA.Historian.Client` under `histsdk/`; browse/metadata + SendEvent broadening merged to `main` + pushed. **`SendEvent` is also amortized** via a **separate, parallel event-session pool** (`Historian:EventSessionPool`, default-on; v8/ECDH auth — kept distinct from the v6 pool), warranted by a GREEN v8 Event-session reuse spike (~10–16×); `ReadEvents` stays per-call / gated (C2). The full offline suite is green on macOS (0 warnings); the env-gated live historian + Galaxy integration suite exercises the amortized path and otherwise skips without a live server. **Part C minimal (pending.md C1/C3a) merged to `main`:** `Int8`/`UInt8` live write types un-gated + live-proven against `wonder-sql-vd03` (in the owned `histsdk/` SDK); the 2023 R2 gRPC interface-version integers recorded as evidence (C3a); `UInt1` attempted but **server-blocked** (the historian accepts `EnsureTags(UInt1)` yet stores a degenerate analog tag) → re-gated fail-closed. **C2 closed won't-fix (2026-06-26):** event reads are server-gated on the 2023 R2 historian over **both** transports — gRPC retrieval-server-gated (0 rows scoped to a managed connection), and the WCF certificate transport + auth DO reach the historian cross-platform (CM_EVENT registers on the `0x501` event connection) but the query still returns 0 rows: the same server-side per-connection row gate. (An earlier note saying "WCF not served on 2023 R2" was a test error vs the reverse tunnel.) Not client-fixable. Reusable SDK wins: `ConnectViaAddress` (WCF Via for tunneled access), `EventReadConnectionModeOverride`. **SQL-path `ReadEvents` (merged 2026-06-26) — the practical workaround for the C2 gate:** event reads now **ship** via the historian's `Runtime.dbo.Events` SQL view (config-gated `RuntimeDb:EventReadsEnabled` + `EventReadMaxRows`), mirroring the SQL live-write path (no COM/native); live-proven streaming real events (incl. an INSQL NOT-NULL `CAST` fix the live test caught) while the native event-query stays gated — so **event reads work** despite the native dead-end. **All of Part C + C2 + SQL-ReadEvents are merged to `origin/main`** (gateway @ `fabab1a`, histsdk @ `f0a1b04`). **Follow-ups merged to `main` (2026-06-27):** `historiangw` **PR #6** — `WriteLiveValues` UTC→server-local timestamp fix (live-validated exact) + a documented `SendEvent` `Source_Object` protocol limitation (`pending.md` C4); the Plan-1 .NET **client lib + packable Contracts** (PR #5); and a C2 cross-ref docs PR (#4). **First consumer (merged):** OtOpcUa **PR #423** (→ `master`) adopts `ZB.MOM.WW.HistorianGateway.Client` as its sole historian read/write backend, retiring its bespoke Wonderware historian driver. **Local Docker (2026-06-29):** `HistorianGateway/docker/` runs the sidecar in a container against the REAL wonder datasources (historian `wonder-sql-vd03:32565` TLS + Galaxy SQL `wonder-app-vd03`/`ZB`), dashboard login + gRPC API-key auth disabled — verified `/health/ready` Healthy. Runtime-only image: publish framework-dependent on the host (authed Gitea feed → no Docker restore), COPY into `aspnet:10.0`; secrets via gitignored `env_file`; isolated compose project `zb-historiangw`; container egress needs the host VPN up. **Client surface (`clients/dotnet/`):** the published `ZB.MOM.WW.HistorianGateway.Client` lib + a `ZB.MOM.WW.HistorianGateway.Client.Cli` smoke CLI (`probe`/`read-raw`/`browse`, JSON out; in the `.slnx`, not packed). **Current: `origin/main` @ `56a70df`** — **AHC-002 string READS landed via native RE from the Mac (PR #17 merged 2026-07-16):** the SDK now parses string historian result-buffer rows into `HistorianSample.StringValue` (a per-row `valueType` discriminator — `1`=numeric→`double`, `3`=string; value = `u32 charLen` + UTF-16LE), and the gateway forwards `string_value` through `HistorianRead.ReadRaw` — **no contract change, no package bump** (the proto already carried the field; mapper already forwarded it); live-proven against 2023 R2. String **tag-create** is now doable via SQL `aaStringTagInsert` (`SqlStringTagProvisioner`, test tooling — single + double byte). String **historical writes** were probed (bounded 6-candidate native blob probe) but **non-convergent** — tag-info resolves a **real GUID + SingleByteString** (unlike the Int1/UInt1 degenerate stub) so writes aren't blocked at resolution, but `BSuccess` is **not an oracle** (all framings optimistically accepted) and none round-tripped → **deferred, needs a native-client write wire capture** (blind guessing can't finish). Eight follow-up issues filed (`historiangw` #18–#25): datatype gaps (native string tag-create #18, string writes #19, DateTime/FileTime #20, Guid/Structure/Event #21, Int1/UInt1 discrete-create #22 [follow-on to #11], DoubleByteString read-verify #23) + **SQL-fallback trackers** (event reads via `Runtime.dbo.Events` #24, live writes via `History`/`v_StringHistory` #25). Also folded a doc-truth reconcile of the "Key structural decisions" bullets (GalaxyRepository `PackageReference`, Contracts/Client 0.3.0 published, OtOpcUa on 0.3.0). Prior: **ArchReview #2 (cycle-2) remediation MERGED 2026-07-14 (PR #12):** all 8 workstreams A–H — store-forward drain honesty (`IsRetryableDeliveryFault`, typed `HistorianOutboxFullException`→`ResourceExhausted`), streaming-lease caps (per-key cap 8→4, `MaxStreamLeaseHold`, bounded `EndQuery` teardown, client `StreamTimeout` default null→10min), a **live idle-out capture (GREEN-A, native `(type 4, code 51)`) that narrowed the session-expiry classifier and closed the stability-F5 tail**, histsdk residue (typed event-send `FromNativeError`, `HistorianProtocolIntegrityException`, IP/host scrub), read-path caps (`MaxAggregateIntervals`/`MaxAtTimeTimestamps`, 5M→`ResourceExhausted`, verify-once-per-connection version gate, batched audit drain), release hygiene (csproj `` sole authority + packed-README/AnyCPU/FQDN/CHANGELOG guards), security/ops (opt-in `ForwardedHeaders` trust list, k8s key-bootstrap runbook, dashboard pool/dead-letter signals), and doc-truth #2 (`Historian:VerifyServerInterfaceVersion` now a real fail-closed key). Verified 0-warn, offline 1388 pass + live suite green vs `wonder-sql-vd03`. **Contracts+Client 0.3.0 PUBLISHED** to the `dohertj2-gitea` feed (2026-07-14; carries the `StreamTimeout` default change) and **OtOpcUa bumped to 0.3.0** (`lmxopcua master @ 7f79cd59`, PR #440 — only consumer delta is the StreamTimeout default; driver tests 103/0). **Boolean historization remapped Int1→Int2 (2026-07-14, `lmxopcua master @ 38e21df2`, PR #442, closes #441):** the R2-06 live gate surfaced that the historian's `Int1` analog-tag creation path is server-degenerate (`EnsureTags(Int1)` stores an unusable stub) — filed gateway-side as `historiangw` #11, documented there as an explicit evidence-based exclusion (PR #16), and routed back to OtOpcUa as issue #441. OtOpcUa's `HistorianTypeMapper` now maps `Boolean → Int2` (0/1 integer); the value-write path already sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips cleanly and a pre-existing Float Boolean tag retypes cleanly (both are supported analog types). Closes the last R2-06 red test → live gate **6/6**; gateway driver unit 103/103. **archreview round-2 deferred gates closed 2026-07-14** (`lmxopcua master @ 30672888`): **R2-04 T13+T15 LIVE-PASSED** (PR #444) — the behavior-affecting S4 primary-gate default-deny verified on a 2-node docker-dev rig (ServiceLevel 250/240; boot-window write rejected `not primary (role unknown)` + denial meter `reason=role-unknown`; steady-state secondary rejected + node reverted; primary write reaches device) plus the in-process `PrimaryGateFailoverTests` (delivered-snapshot drives the gate); and the pre-existing red S7.Cli phrase-scan test fixed (PR #443, restored a stripped rationale comment). Round-2 remainder is now only infra-gated: R2-08 live-GLAuth outage, R2-01 S7 SYN-blackhole. **The full `*.IntegrationTests` sweep is now COMPLETE (2026-07-15, `lmxopcua master @ 152a5645`):** ran all 10 driver + server integration suites serially (on the Mac against native fixtures on the `10.100.0.35` Docker host), then closed all 7 resulting follow-ups (PRs #446–#452) — **every driver + server integration suite is verified GREEN with ZERO OtOpcUa production regressions;** every original "failure" was fixture/harness/test rot, not product. Highlights of cross-repo relevance: (a) **the Host integration harness's LDAP swapped from the retired `bitnami/openldap:2.6` to GLAuth** (PR #451) — the harness now runs its own ephemeral GLAuth container on `:3894` (seeded by `tests/.../Host.IntegrationTests/glauth/config.toml`, distinct from the shared dev GLAuth on `:3893`), unifying the whole family's opt-in real-LDAP path on GLAuth and removing the lone OpenLDAP outlier; (b) the deploy-E2E "amd64-emulated-SQL timing" complaint was **misdiagnosed** — those tests had only ever run against the EF in-memory provider (no FK enforcement) and lacked the `ServerCluster`/`ClusterNode` seed the real-SQL `NodeDeploymentState` FK requires; fixed with a harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST`/`_LDAP_HOST` overrides so the fixtures run on the native-amd64 Docker host (PR #452). See `archreview/plans/INTEGRATION-SWEEP-STATUS.md` for the full record. Cycle-2 gated tail: CI first-green-run (needs a Gitea act_runner), the histsdk protobuf/Grpc.Tools major bump (03-CC-8, blocked on migrating off obsolete `GrpcWebHandler.HttpVersion`), and the 1.0.0 TLS-default flip. The prior cycle-1 **ArchReview remediation is fully merged** (Now + Next tiers 2026-07-09 @ `508b9c3`; **Later tier 2026-07-13** via a coordinated two-PR merge, `12385c8` + `6a24a30`). Off a 2026-07-08 six-report architecture review (`archreview/`) turned into 8 workstream plans (`docs/plans/2026-07-09-archreview-*`; tracker `...-remediation-tracking.md`), shipped clean under `TreatWarningsAsErrors`: a hard-fail `ResilienceComposition` validator (StoreForward+Redundancy both-on), corruption-tolerant outbox + dead-letter/overflow knobs, `/auth/login` rate-limit, server-side `max_values` cap, a headless `apikey` CLI, histsdk parser hardening, Gitea Actions CI + warnings-as-errors on all trees, pool observability (lease-wait/waiter/counters) + session-pool health probe, and a typed histsdk fault taxonomy with type-based classification + session churn-guard. **Contracts+Client 0.2.0 published** to the `dohertj2-gitea` feed (2026-07-09; Int1 proto3-optional, opc_quality doc, CA-chain pin). **Cross-repo 0.2.0/G-2 chain COMPLETE:** OtOpcUa bumped to Client+Contracts `0.2.0` (2026-07-10, `master @ f6eaa267`; M3 `opc_quality` gate clear), and the shared `ZB.MOM.WW.Auth.ApiKeys` gained `ExpiresUtc` at `0.1.4` — consumed by HistorianGateway (`apikey create --expires` + dashboard Expires column) and mxaccessgw (verifier enforcement). **Later tier (roadmap 16–21) MERGED to `main` 2026-07-13** (coordinated two-PR merge; both branches now deleted): **PR #8** (`archreview-later-21-and-maxsessions`, merge `12385c8`) = Later-21 (audit off the mutating-RPC critical path + `WriteLiveValues` SQL batching + composed-failure/shutdown tests) + the **MaxSessions 4→8 raise** (live concurrent-session spike GREEN); **PR #9** (`later-16-page-through-streaming`, merge `6a24a30`) = Later-16 **ReadRaw page-through streaming** (streams per server result buffer; pool holds the lease across the enumeration; new `Historian:ReadLimits:AllowStreamingBeyondCeiling`), Later-17 **redundancy honest-hardening** (exposes the `Redundancy:WriteAcknowledgement`/`WriteFanout`/`FailureThreshold` ack policy the docs already claimed), Later-18 **per-key concurrency cap** (`ApiKeys:MaxConcurrentCallsPerKey` → gRPC `ResourceExhausted`) + api-key-rotation runbook, Later-19 CHANGELOG-discipline guard test, Later-20 **K8s manifests + hardened prod Dockerfile + operator runbooks** (`deploy/k8s/`, `docker/Dockerfile.production`, `docs/runbooks/`). The two PRs conflicted on `CLAUDE.md`/tracking/`ValidatorTests`, resolved as the union (MaxSessions lands at **8**). Live-proven vs `wonder-sql-vd03`; offline gateway 729 pass / client 140 pass / 0 warnings. **Only remaining archreview item: the secure-by-default TLS-default flip, deferred to 1.0.0** (pre-announced in the client 0.2.0 CHANGELOG). (Prior tip `93c6051` = the 2026-07-07 CommentChecker XML-doc sweep.) |