From 58c2e14c3845b4e8f196caa46e62fc63b4091786 Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Thu, 13 Aug 2026 08:49:25 -0400
Subject: [PATCH] docs(index): Auth 0.2.1 adopted by all four apps;
 FallbackServers live-gated on HG + ScadaBridge rig

---
 CLAUDE.md                        |  6 +++---
 infra/glauth-redundant/README.md | 15 +++++++++++----
 ldap_setup.md                    | 10 +++++++---
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 5758aa7..d459049 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -141,7 +141,7 @@ each project's **code-verified current state**, and the **gaps** between. See
 
 | Component | Status | Goal | Design | Implementation |
 |---|---|---|---|---|
-| Auth (login / identity / authz) | Adopted (**feed now carries `0.2.1`**; all four apps still pin `0.1.5` = `0.1.4` + the SQLitePCLRaw transitive security pin; merged to **local default** main/master + **pushed to origin** (gitea)). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`. **Every app is still exposed until it bumps** — see the Auth narrative below; app pin-bump is a follow-up. | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
+| Auth (login / identity / authz) | Adopted (**feed carries `0.2.1`, and ALL FOUR apps pin `0.2.1` as of 2026-08-13**: OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway `main @ 1cb14d2`, HistorianGateway `e42ff31` on its `dashboard-ui-sweep-theme-0.4.1` branch — each built + tested green, commits local / NOT pushed). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
 | UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.4.1`). **`0.4.0` (2026-08-11) upstreamed the Bootstrap `.btn`/`.btn-sm` sizing block** that all **five** consumers had independently written into their own stylesheets (Bootstrap hard-codes `--bs-btn-font-size: 1rem`, oversized against the kit's 0.9rem body) — sized by `--bs-btn-*` variable override **only**, never by redeclaring box properties, because `.btn-group`'s seam/radius machinery reads those same variables and a literal `font-size` at equal specificity silently flattens `.btn-sm` into a padding-only difference (a real defect found in mxgw's local copy). Same release added the `rail-btn-block` full-width rail-foot modifier; **`0.4.1` fixes it** — 0.4.0 shipped `width: auto`, which fills for an `<a>` but shrink-wraps a `<button>` (measured 114px short in the same rail), now `calc(100% - 1.2rem)` to clear `.rail-btn`'s side margins. **Consumers are five, not four** — the ignitionoee router joined. **All five consumers are on `0.4.1` and all five are PUSHED** (2026-08-11): ignitionoee `main @ 38aa5c1`, OtOpcUa `master @ e82881c1`, mxgw `main @ 0a9715d`. The two whose sweeps were not deploy-ready pushed to **branches**, deliberately leaving their default branch untouched: HistorianGateway `dashboard-ui-sweep-theme-0.4.1 @ df7b879` (`main` stays `e9c31da`) and ScadaBridge `ui/central-ui-cleanup-sweep @ 9e243493` (`main` was then `b6f383a2`). **ScadaBridge's caveat is now CLOSED and its sweep is merged, deployed and live** (2026-08-11): the branch landed on `main`, and the two follow-ups `d14e0ee4` + `3eb7df74` fixed exactly the four detail-modal surfaces the caveat named — the modal is now gated on **user intent (the held id)**, never on the row still resolving, so a background refresh that drops the row leaves it mounted and says why the summary is missing instead of unmounting the subtree and disposing Close's own handler mid-click. That disposal *was* the intermittent `GetRequiredEventBindingEntry`, and `NotificationReportDetailModalTests.Modal_StaysOpen_WhenItsRowLeavesThePage` now pins it. **Deployed to wonder-app-vd03: OtOpcUa `e82881c1`, and ScadaBridge `3eb7df74` (2026-08-11 — theme.css/layout.css/nav-state.js served from `_content/ZB.MOM.WW.Theme` and referenced by the pages; all six changed routes 200). HistorianGateway is still `e9c31da` = 0.3.1 — its 0.4.1 remains on the branch.** **On bump, TRIM the local `.btn` block, don't delete it** — four of the five were pure sizing, but mxgw's also carried `border-radius`/`font-weight`/`white-space` that a blind delete would have dropped. | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
 | Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
 | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
@@ -166,12 +166,12 @@ committed + reviewed, then **fast-forward-merged into the repo's local default (
 (gitea) on 2026-06-03** (in sync; the `feat/*` branches kept locally as history). Cutover: shared `Auth.Ldap`,
 `Auth.ApiKeys` (ScadaBridge inbound fully re-architected to the keyId/Bearer model), `IGroupRoleMapper<TRole>` seam,
 `Transport`-enum config, canonical `ZbClaimTypes`/`ZbCookieDefaults`, unified dev base DN `dc=zb,dc=local`, and the
-canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins (all verified 2026-08-07 against each repo's `Directory.Packages.props`, all on origin): **all four apps pin `0.1.5`** — MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
+canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins: **all four apps pin `0.2.1` as of 2026-08-13** (commits listed in the table row above; local, not yet pushed). Prior history — all four pinned `0.1.5` (verified 2026-08-07): MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
 `docs/plans/2026-06-02-auth-audit-normalization*.md`.
 Build/test from `ZB.MOM.WW.Auth/`: `dotnet test`. Consumer matrix: OtOpcUa → Abstractions+Ldap+AspNetCore;
 MxAccessGateway & ScadaBridge → all four (ApiKeys not used by OtOpcUa).
 **`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger<LdapAuthService>` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up.
-**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping — and it is a four-minor jump that also lands `0.2.0`'s `FallbackServers`.
+**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`).
 
 **`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real** `LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups '<pat>'` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain.
 
diff --git a/infra/glauth-redundant/README.md b/infra/glauth-redundant/README.md
index 3d92f40..bc4360b 100644
--- a/infra/glauth-redundant/README.md
+++ b/infra/glauth-redundant/README.md
@@ -31,10 +31,17 @@ Ldap__Transport: None
 Ldap__AllowInsecure: "true"
 ```
 
-First wired consumer: `HistorianGateway/docker/docker-compose.yml` (Auth pin
-`0.2.1`, dashboard login enabled). Note its Production-env container needs
-`Serilog__WriteTo__0__Name: Console` or it logs **nothing** — the drill's
-failover lines would be invisible.
+Wired consumers (both drilled PASS 2026-08-13):
+
+- `HistorianGateway/docker/docker-compose.yml` (Auth `0.2.1`, dashboard login
+  enabled). Note its Production-env container needs
+  `Serilog__WriteTo__0__Name: Console` or it logs **nothing** — the drill's
+  failover lines would be invisible.
+- ScadaBridge docker rig (`docker/central-node-{a,b}/appsettings.Central.json`,
+  commit `b71fbae3`) — this pair REPLACED the shared `10.100.0.35` GLAuth there,
+  whose serviceaccount password rotation (SEC-36) had silently broken rig login.
+  ScadaBridge's `AuthFlowTests` also target `localhost:3893` and pass 5/5 with
+  the pair up (they skip when it is down).
 
 ## Failover drill (verified PASS 2026-08-13 against HistorianGateway)
 
diff --git a/ldap_setup.md b/ldap_setup.md
index c1e8539..a050606 100644
--- a/ldap_setup.md
+++ b/ldap_setup.md
@@ -242,9 +242,13 @@ Three gaps to settle before this is a design rather than a sketch:
 
 **Found and fixed 2026-08-12.** All four Auth packages are **published to the Gitea feed at
 `0.2.1`** and restore-verified from a scratch consumer (the assembly restored *from the
-feed* carries the fix; `0.2.0` does not). **The four apps still pin `0.1.5`, so nothing is
-fixed for them until each bumps** — a pin bump is the remaining step, and until it happens
-every app still fails a domain-root search with `ServiceAccountBindFailed`.
+feed* carries the fix; `0.2.0` does not). **ADOPTED by all four apps 2026-08-13** —
+OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway
+`main @ 1cb14d2`, HistorianGateway `e42ff31` (on its `dashboard-ui-sweep-theme-0.4.1`
+branch) — each built + tested green; commits are local, not yet pushed. `FallbackServers`
+failover was live-gated the same day against the local redundant GLAuth pair
+([`infra/glauth-redundant/`](infra/glauth-redundant/)): HistorianGateway dashboard login
+drill PASS (sticky failover both directions).
 
 Authenticating with `SearchBase = DC=nam,DC=zimmer,DC=com` fails on **both DCs** with
 `ServiceAccountBindFailed` in ~546 ms. It is not connectivity, not the service account,