diff --git a/CLAUDE.md b/CLAUDE.md index 33d5352..c1fa5e7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-th | Project | Location | Stack | Repo | Summary | |---|---|---|---|---| -| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient), a **read-only `Sql` DB-poll driver** (polls SQL Server tables/views on an interval → OPC UA variables; `connectionStringRef` env resolution, schema-walk address picker, live-gated on docker-dev — merged to `master` 2026-07-24), a **read-only `Mqtt` broker-subscribe driver with Sparkplug B** (one MQTTnet-5 client per driver, TLS+auth, hand-rolled reconnect; **Plain** mode binds a tag to a concrete topic, **SparkplugB** mode decodes vendored Eclipse-Tahu protobuf under one `spBv1.0/{groupId}/#` subscription and binds by the `group/edgeNode/device?/metric` tuple — birth/alias/seq-gap/rebirth state machine, death→STALE, a birth-driven browse picker with a scoped Request-rebirth NCMD; **write-through is deferred, every MQTT node is read-only**; live-gated on docker-dev against a Mosquitto TLS+auth broker + a project-owned C# Sparkplug edge-node simulator on `10.100.0.35:8883` — **merged to `master` 2026-07-27, merge `c3a2b0f7`, pushed to origin**), **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=;s=…` form + doc sweep. The binding re-author (legacy `ns=2` collides with v3 `raw`), native-alarm dedup across the raw+uns fan-out, and UNS-bound→RawPath alarm routing are DEFERRED to a live v3 rig — ScadaBridge #14 stays OPEN. Scope: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is still open** (the pathology needs 1-vs-1; docker-dev is one six-node mesh) and is deferred to the per-cluster mesh work. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **Phases 0a/0b/1/2/3/4/5/6 done + merged to `master` (Phase 6 pushed to origin 2026-07-24, tip `2839deb5`; Phase 5 was `35552a96`), 7 not started.** **Phase 2 (merge `6eaa81ca`)** added a `ClusterClient` command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership). **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — code done+reviewed, merged to `master` + pushed 2026-07-24, tip `2839deb5`; its live gate DEFERRED by owner choice "merge now, gate after")** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`; the split now makes the auto-down 1-vs-1 crash-the-oldest live gate (open since Phase 0a) finally testable, deferred to Phase 7. | +| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient), a **read-only `Sql` DB-poll driver** (polls SQL Server tables/views on an interval → OPC UA variables; `connectionStringRef` env resolution, schema-walk address picker, live-gated on docker-dev — merged to `master` 2026-07-24), a **read-only `Mqtt` broker-subscribe driver with Sparkplug B** (one MQTTnet-5 client per driver, TLS+auth, hand-rolled reconnect; **Plain** mode binds a tag to a concrete topic, **SparkplugB** mode decodes vendored Eclipse-Tahu protobuf under one `spBv1.0/{groupId}/#` subscription and binds by the `group/edgeNode/device?/metric` tuple — birth/alias/seq-gap/rebirth state machine, death→STALE, a birth-driven browse picker with a scoped Request-rebirth NCMD; **write-through is deferred, every MQTT node is read-only**; live-gated on docker-dev against a Mosquitto TLS+auth broker + a project-owned C# Sparkplug edge-node simulator on `10.100.0.35:8883` — **merged to `master` 2026-07-27, merge `c3a2b0f7`, pushed to origin**), a **read-only `MTConnect` Agent driver** (one hand-rolled `System.Xml.Linq` HTTP/XML client per driver against a vendor-neutral MTConnect Agent — `/probe` builds the device model, `/current` primes, and a `multipart/x-mixed-replace` `/sample` long-poll pump feeds `ISubscribable.OnDataChange` with ring-buffer sequence-gap re-baselining and a heartbeat watchdog; `UNAVAILABLE` → `BadNoCommunication`; the TrakHound MTConnect.NET dependency was evaluated and **dropped** — neither pinned package can parse an MTConnect document or frame the sample stream socket-free; browse comes free from the Wave-0 universal discovery browser, so the driver ships no browser project; **write-back is deferred, every MTConnect node is read-only**; live-gated on an isolated docker-dev stack against a real `mtconnect/agent:2.7.0.12` + SHDR-adapter fixture on `10.100.0.35:5000` — **merged to `master` 2026-07-27, PR #506, merge `90bdaa44`, pushed to origin**), **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=;s=…` form + doc sweep. The binding re-author (legacy `ns=2` collides with v3 `raw`), native-alarm dedup across the raw+uns fan-out, and UNS-bound→RawPath alarm routing are DEFERRED to a live v3 rig — ScadaBridge #14 stays OPEN. Scope: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is still open** (the pathology needs 1-vs-1; docker-dev is one six-node mesh) and is deferred to the per-cluster mesh work. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **Phases 0a/0b/1/2/3/4/5/6 done + merged to `master` (Phase 6 pushed to origin 2026-07-24, tip `2839deb5`; Phase 5 was `35552a96`), 7 not started.** **Phase 2 (merge `6eaa81ca`)** added a `ClusterClient` command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership). **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — code done+reviewed, merged to `master` + pushed 2026-07-24, tip `2839deb5`; its live gate DEFERRED by owner choice "merge now, gate after")** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`; the split now makes the auto-down 1-vs-1 crash-the-oldest live gate (open since Phase 0a) finally testable, deferred to Phase 7. | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** | | **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. | | **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib (consumed as a Gitea-feed package). No COM, no x86 worker. **Dev:** two plaintext endpoints from `appsettings.Development.json` — dashboard on `:5220` (HTTP/1.1), gRPC h2c on `:5221`. **Production:** single `Kestrel:Endpoints:Https` endpoint with `Protocols: Http1AndHttp2` multiplexes dashboard + gRPC over one TLS port (ALPN); warn-only if no TLS endpoint configured (valid behind a reverse proxy / Kubernetes ingress; the warn predicate covers any non-Development environment, i.e. Production + Staging). In a non-Development environment the gateway also logs warn-only **production-readiness** checks (pending.md D2/D3) — relative runtime-artifact paths + secret hygiene (`ApiKeys:Mode=Disabled`, empty/dev-placeholder pepper, dev-placeholder LDAP password). **Owns `AVEVA.Historian.Client`** — imported into the repo at `histsdk/` (subtree, folded in with history 2026-06-28; the old "vendored / re-vendor from upstream" model is **retired** — `histsdk/**` is ordinary owned source, edited in place; the gateway builds it directly). Store-forward uses a crash-safe FasterLog append-only outbox (`Microsoft.FASTER.Core` 2.6.5; `CommitMode` PerEntry/Periodic), not SQLite. **Handshake amortization (pending.md A1) done + live-validated** — a default-on leased-session pool (`Historian:SessionPool`) reuses pre-authenticated sessions across reads/writes/status ops/tag-browse/metadata (~4.7× measured; probe and blocks stay per-call), with a `<~15 s` keepalive + reactive re-auth, surfaced via a `PooledHistorianClient` facade so services are unchanged; the `HistorianSession` primitive lives in the owned `AVEVA.Historian.Client` under `histsdk/`; browse/metadata + SendEvent broadening merged to `main` + pushed. **`SendEvent` is also amortized** via a **separate, parallel event-session pool** (`Historian:EventSessionPool`, default-on; v8/ECDH auth — kept distinct from the v6 pool), warranted by a GREEN v8 Event-session reuse spike (~10–16×); `ReadEvents` stays per-call / gated (C2). The full offline suite is green on macOS (0 warnings); the env-gated live historian + Galaxy integration suite exercises the amortized path and otherwise skips without a live server. **Part C minimal (pending.md C1/C3a) merged to `main`:** `Int8`/`UInt8` live write types un-gated + live-proven against `wonder-sql-vd03` (in the owned `histsdk/` SDK); the 2023 R2 gRPC interface-version integers recorded as evidence (C3a); `UInt1` attempted but **server-blocked** (the historian accepts `EnsureTags(UInt1)` yet stores a degenerate analog tag) → re-gated fail-closed. **C2 closed won't-fix (2026-06-26):** event reads are server-gated on the 2023 R2 historian over **both** transports — gRPC retrieval-server-gated (0 rows scoped to a managed connection), and the WCF certificate transport + auth DO reach the historian cross-platform (CM_EVENT registers on the `0x501` event connection) but the query still returns 0 rows: the same server-side per-connection row gate. (An earlier note saying "WCF not served on 2023 R2" was a test error vs the reverse tunnel.) Not client-fixable. Reusable SDK wins: `ConnectViaAddress` (WCF Via for tunneled access), `EventReadConnectionModeOverride`. **SQL-path `ReadEvents` (merged 2026-06-26) — the practical workaround for the C2 gate:** event reads now **ship** via the historian's `Runtime.dbo.Events` SQL view (config-gated `RuntimeDb:EventReadsEnabled` + `EventReadMaxRows`), mirroring the SQL live-write path (no COM/native); live-proven streaming real events (incl. an INSQL NOT-NULL `CAST` fix the live test caught) while the native event-query stays gated — so **event reads work** despite the native dead-end. **All of Part C + C2 + SQL-ReadEvents are merged to `origin/main`** (gateway @ `fabab1a`, histsdk @ `f0a1b04`). **Follow-ups merged to `main` (2026-06-27):** `historiangw` **PR #6** — `WriteLiveValues` UTC→server-local timestamp fix (live-validated exact) + a documented `SendEvent` `Source_Object` protocol limitation (`pending.md` C4); the Plan-1 .NET **client lib + packable Contracts** (PR #5); and a C2 cross-ref docs PR (#4). **First consumer (merged):** OtOpcUa **PR #423** (→ `master`) adopts `ZB.MOM.WW.HistorianGateway.Client` as its sole historian read/write backend, retiring its bespoke Wonderware historian driver. **Local Docker (2026-06-29):** `HistorianGateway/docker/` runs the sidecar in a container against the REAL wonder datasources (historian `wonder-sql-vd03:32565` TLS + Galaxy SQL `wonder-app-vd03`/`ZB`), dashboard login + gRPC API-key auth disabled — verified `/health/ready` Healthy. Runtime-only image: publish framework-dependent on the host (authed Gitea feed → no Docker restore), COPY into `aspnet:10.0`; secrets via gitignored `env_file`; isolated compose project `zb-historiangw`; container egress needs the host VPN up. **Client surface (`clients/dotnet/`):** the published `ZB.MOM.WW.HistorianGateway.Client` lib + a `ZB.MOM.WW.HistorianGateway.Client.Cli` smoke CLI (`probe`/`read-raw`/`browse`, JSON out; in the `.slnx`, not packed). **Current: `origin/main` @ `56a70df`** — **AHC-002 string READS landed via native RE from the Mac (PR #17 merged 2026-07-16):** the SDK now parses string historian result-buffer rows into `HistorianSample.StringValue` (a per-row `valueType` discriminator — `1`=numeric→`double`, `3`=string; value = `u32 charLen` + UTF-16LE), and the gateway forwards `string_value` through `HistorianRead.ReadRaw` — **no contract change, no package bump** (the proto already carried the field; mapper already forwarded it); live-proven against 2023 R2. String **tag-create** is now doable via SQL `aaStringTagInsert` (`SqlStringTagProvisioner`, test tooling — single + double byte). String **historical writes** were probed (bounded 6-candidate native blob probe) but **non-convergent** — tag-info resolves a **real GUID + SingleByteString** (unlike the Int1/UInt1 degenerate stub) so writes aren't blocked at resolution, but `BSuccess` is **not an oracle** (all framings optimistically accepted) and none round-tripped → **deferred, needs a native-client write wire capture** (blind guessing can't finish). Eight follow-up issues filed (`historiangw` #18–#25): datatype gaps (native string tag-create #18, string writes #19, DateTime/FileTime #20, Guid/Structure/Event #21, Int1/UInt1 discrete-create #22 [follow-on to #11], DoubleByteString read-verify #23) + **SQL-fallback trackers** (event reads via `Runtime.dbo.Events` #24, live writes via `History`/`v_StringHistory` #25). Also folded a doc-truth reconcile of the "Key structural decisions" bullets (GalaxyRepository `PackageReference`, Contracts/Client 0.3.0 published, OtOpcUa on 0.3.0). Prior: **ArchReview #2 (cycle-2) remediation MERGED 2026-07-14 (PR #12):** all 8 workstreams A–H — store-forward drain honesty (`IsRetryableDeliveryFault`, typed `HistorianOutboxFullException`→`ResourceExhausted`), streaming-lease caps (per-key cap 8→4, `MaxStreamLeaseHold`, bounded `EndQuery` teardown, client `StreamTimeout` default null→10min), a **live idle-out capture (GREEN-A, native `(type 4, code 51)`) that narrowed the session-expiry classifier and closed the stability-F5 tail**, histsdk residue (typed event-send `FromNativeError`, `HistorianProtocolIntegrityException`, IP/host scrub), read-path caps (`MaxAggregateIntervals`/`MaxAtTimeTimestamps`, 5M→`ResourceExhausted`, verify-once-per-connection version gate, batched audit drain), release hygiene (csproj `` sole authority + packed-README/AnyCPU/FQDN/CHANGELOG guards), security/ops (opt-in `ForwardedHeaders` trust list, k8s key-bootstrap runbook, dashboard pool/dead-letter signals), and doc-truth #2 (`Historian:VerifyServerInterfaceVersion` now a real fail-closed key). Verified 0-warn, offline 1388 pass + live suite green vs `wonder-sql-vd03`. **Contracts+Client 0.3.0 PUBLISHED** to the `dohertj2-gitea` feed (2026-07-14; carries the `StreamTimeout` default change) and **OtOpcUa bumped to 0.3.0** (`lmxopcua master @ 7f79cd59`, PR #440 — only consumer delta is the StreamTimeout default; driver tests 103/0). **Boolean historization remapped Int1→Int2 (2026-07-14, `lmxopcua master @ 38e21df2`, PR #442, closes #441):** the R2-06 live gate surfaced that the historian's `Int1` analog-tag creation path is server-degenerate (`EnsureTags(Int1)` stores an unusable stub) — filed gateway-side as `historiangw` #11, documented there as an explicit evidence-based exclusion (PR #16), and routed back to OtOpcUa as issue #441. OtOpcUa's `HistorianTypeMapper` now maps `Boolean → Int2` (0/1 integer); the value-write path already sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips cleanly and a pre-existing Float Boolean tag retypes cleanly (both are supported analog types). Closes the last R2-06 red test → live gate **6/6**; gateway driver unit 103/103. **archreview round-2 deferred gates closed 2026-07-14** (`lmxopcua master @ 30672888`): **R2-04 T13+T15 LIVE-PASSED** (PR #444) — the behavior-affecting S4 primary-gate default-deny verified on a 2-node docker-dev rig (ServiceLevel 250/240; boot-window write rejected `not primary (role unknown)` + denial meter `reason=role-unknown`; steady-state secondary rejected + node reverted; primary write reaches device) plus the in-process `PrimaryGateFailoverTests` (delivered-snapshot drives the gate); and the pre-existing red S7.Cli phrase-scan test fixed (PR #443, restored a stripped rationale comment). Round-2 remainder is now only infra-gated: R2-08 live-GLAuth outage, R2-01 S7 SYN-blackhole. **The full `*.IntegrationTests` sweep is now COMPLETE (2026-07-15, `lmxopcua master @ 152a5645`):** ran all 10 driver + server integration suites serially (on the Mac against native fixtures on the `10.100.0.35` Docker host), then closed all 7 resulting follow-ups (PRs #446–#452) — **every driver + server integration suite is verified GREEN with ZERO OtOpcUa production regressions;** every original "failure" was fixture/harness/test rot, not product. Highlights of cross-repo relevance: (a) **the Host integration harness's LDAP swapped from the retired `bitnami/openldap:2.6` to GLAuth** (PR #451) — the harness now runs its own ephemeral GLAuth container on `:3894` (seeded by `tests/.../Host.IntegrationTests/glauth/config.toml`, distinct from the shared dev GLAuth on `:3893`), unifying the whole family's opt-in real-LDAP path on GLAuth and removing the lone OpenLDAP outlier; (b) the deploy-E2E "amd64-emulated-SQL timing" complaint was **misdiagnosed** — those tests had only ever run against the EF in-memory provider (no FK enforcement) and lacked the `ServerCluster`/`ClusterNode` seed the real-SQL `NodeDeploymentState` FK requires; fixed with a harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST`/`_LDAP_HOST` overrides so the fixtures run on the native-amd64 Docker host (PR #452). See `archreview/plans/INTEGRATION-SWEEP-STATUS.md` for the full record. Cycle-2 gated tail: CI first-green-run (needs a Gitea act_runner), the histsdk protobuf/Grpc.Tools major bump (03-CC-8, blocked on migrating off obsolete `GrpcWebHandler.HttpVersion`), and the 1.0.0 TLS-default flip. The prior cycle-1 **ArchReview remediation is fully merged** (Now + Next tiers 2026-07-09 @ `508b9c3`; **Later tier 2026-07-13** via a coordinated two-PR merge, `12385c8` + `6a24a30`). Off a 2026-07-08 six-report architecture review (`archreview/`) turned into 8 workstream plans (`docs/plans/2026-07-09-archreview-*`; tracker `...-remediation-tracking.md`), shipped clean under `TreatWarningsAsErrors`: a hard-fail `ResilienceComposition` validator (StoreForward+Redundancy both-on), corruption-tolerant outbox + dead-letter/overflow knobs, `/auth/login` rate-limit, server-side `max_values` cap, a headless `apikey` CLI, histsdk parser hardening, Gitea Actions CI + warnings-as-errors on all trees, pool observability (lease-wait/waiter/counters) + session-pool health probe, and a typed histsdk fault taxonomy with type-based classification + session churn-guard. **Contracts+Client 0.2.0 published** to the `dohertj2-gitea` feed (2026-07-09; Int1 proto3-optional, opc_quality doc, CA-chain pin). **Cross-repo 0.2.0/G-2 chain COMPLETE:** OtOpcUa bumped to Client+Contracts `0.2.0` (2026-07-10, `master @ f6eaa267`; M3 `opc_quality` gate clear), and the shared `ZB.MOM.WW.Auth.ApiKeys` gained `ExpiresUtc` at `0.1.4` — consumed by HistorianGateway (`apikey create --expires` + dashboard Expires column) and mxaccessgw (verifier enforcement). **Later tier (roadmap 16–21) MERGED to `main` 2026-07-13** (coordinated two-PR merge; both branches now deleted): **PR #8** (`archreview-later-21-and-maxsessions`, merge `12385c8`) = Later-21 (audit off the mutating-RPC critical path + `WriteLiveValues` SQL batching + composed-failure/shutdown tests) + the **MaxSessions 4→8 raise** (live concurrent-session spike GREEN); **PR #9** (`later-16-page-through-streaming`, merge `6a24a30`) = Later-16 **ReadRaw page-through streaming** (streams per server result buffer; pool holds the lease across the enumeration; new `Historian:ReadLimits:AllowStreamingBeyondCeiling`), Later-17 **redundancy honest-hardening** (exposes the `Redundancy:WriteAcknowledgement`/`WriteFanout`/`FailureThreshold` ack policy the docs already claimed), Later-18 **per-key concurrency cap** (`ApiKeys:MaxConcurrentCallsPerKey` → gRPC `ResourceExhausted`) + api-key-rotation runbook, Later-19 CHANGELOG-discipline guard test, Later-20 **K8s manifests + hardened prod Dockerfile + operator runbooks** (`deploy/k8s/`, `docker/Dockerfile.production`, `docs/runbooks/`). The two PRs conflicted on `CLAUDE.md`/tracking/`ValidatorTests`, resolved as the union (MaxSessions lands at **8**). Live-proven vs `wonder-sql-vd03`; offline gateway 729 pass / client 140 pass / 0 warnings. **Only remaining archreview item: the secure-by-default TLS-default flip, deferred to 1.0.0** (pre-announced in the client 0.2.0 CHANGELOG). (Prior tip `93c6051` = the 2026-07-07 CommentChecker XML-doc sweep.) | @@ -143,7 +143,7 @@ each project's **code-verified current state**, and the **gaps** between. See |---|---|---|---|---| | Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | -| Health (readiness / liveness / active-node) | **Adopted** (lib `0.1.0`, on the feed; **all 4 apps** reference it + wire `MapZbHealth` — verified 2026-07-18; the prior "not yet adopted" claim was stale) | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | +| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`/`0.2.0`/`0.2.1`; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. Both app branches are **committed but NOT pushed or merged**. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | diff --git a/ZB.MOM.WW.Health/CLAUDE.md b/ZB.MOM.WW.Health/CLAUDE.md index 10b2633..a9e1e08 100644 --- a/ZB.MOM.WW.Health/CLAUDE.md +++ b/ZB.MOM.WW.Health/CLAUDE.md @@ -4,7 +4,7 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGate The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic. -**Built at 0.1.0. NOT yet adopted by the three apps.** Adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. +**Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`. --- @@ -13,7 +13,7 @@ The library normalizes the three-tier health endpoint convention (`/health/ready | Package | Responsibilities | Key Dependencies | |---|---|---| | `ZB.MOM.WW.Health` | Core tier convention, `MapZbHealth` extension, canonical JSON writer (`ZbHealthWriter`), `IActiveNodeGate` seam, `GrpcDependencyHealthCheck` reachability probe, tier-tag constants (`ZbHealthTags`). No Akka or EF dependency. | `Microsoft.AspNetCore.App` (framework ref), `Grpc.Net.Client` | -| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`), `ActiveNodeHealthCheck` with an optional role filter, and `AkkaActiveNodeGate` that backs `IActiveNodeGate` from cluster member state. | `ZB.MOM.WW.Health`, `Akka.Cluster` | +| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`); `ClusterActiveNode` — the family-wide **oldest-`Up`-member** rule, optionally role-scoped; `ActiveNodeHealthCheck` over it (role-preference list, `activeRole`/`activeNode` in the result `data`); and `AkkaActiveNodeGate`, which backs `IActiveNodeGate` from the same rule. | `ZB.MOM.WW.Health`, `Akka.Cluster` | | `ZB.MOM.WW.Health.EntityFrameworkCore` | `DatabaseHealthCheck` — resolves `IDbContextFactory` when registered, else a scoped `TContext`; pool-safe. Default probe: `CanConnectAsync`. Optional `ProbeQuery` delegate for query-based validation. | `ZB.MOM.WW.Health`, `Microsoft.EntityFrameworkCore` | --- @@ -46,14 +46,19 @@ dotnet test ZB.MOM.WW.Health.slnx dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts ``` -All three test assemblies run offline: +All three test assemblies run offline. The Akka suite forms real in-process clusters on loopback: a +single-node one for the cluster-data payload, and a **two-node one built so the oldest member is not +the lowest-addressed one** — the only condition under which age- and leader-ordering disagree, and so +the only fixture that can catch a leader-based active-node regression. It binds fixed ports +19540/19541 (distinct from OtOpcUa's equivalent fixture) and carries a sanity check that the +divergence actually occurred, so the assertions cannot pass for the wrong reason: | Assembly | Tests | |---|---| -| `ZB.MOM.WW.Health.Tests` | 20 | -| `ZB.MOM.WW.Health.Akka.Tests` | 32 | +| `ZB.MOM.WW.Health.Tests` | 25 | +| `ZB.MOM.WW.Health.Akka.Tests` | 45 | | `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 | -| **Total** | **58** | +| **Total** | **76** | `GeneratePackageOnBuild` is off — pack explicitly with the command above. @@ -61,7 +66,7 @@ All three test assemblies run offline: ## Status -Built at **0.1.0** and published to the Gitea NuGet feed. **Not yet adopted** by the three apps — adoption is tracked in the component backlog: +Built at **0.3.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog: - `~/Desktop/scadaproj/components/health/GAPS.md` — adoption order, effort, and risk diff --git a/ZB.MOM.WW.Health/Directory.Build.props b/ZB.MOM.WW.Health/Directory.Build.props index 4b5ab79..0985c8f 100644 --- a/ZB.MOM.WW.Health/Directory.Build.props +++ b/ZB.MOM.WW.Health/Directory.Build.props @@ -5,7 +5,7 @@ enable enable latest - 0.1.0 + 0.3.0 true + + net10.0 + enable + enable + latest + + + + + true + latest + true + true + + + + + false + + + diff --git a/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.slnx b/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.slnx new file mode 100644 index 0000000..c35418c --- /dev/null +++ b/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.slnx @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/ZB.MOM.WW.Overview/docker/.dockerignore b/ZB.MOM.WW.Overview/docker/.dockerignore new file mode 100644 index 0000000..f9aff11 --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/.dockerignore @@ -0,0 +1,2 @@ +**/*.md +**/.DS_Store diff --git a/ZB.MOM.WW.Overview/docker/Dockerfile b/ZB.MOM.WW.Overview/docker/Dockerfile new file mode 100644 index 0000000..be08112 --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/Dockerfile @@ -0,0 +1,29 @@ +# Runtime-only image for the ZB.MOM.WW.Overview dashboard. +# +# The app is published FRAMEWORK-DEPENDENT on the host (which holds the Gitea NuGet feed +# credentials in ~/.nuget) into ./publish, then copied in here. Docker performs NO restore and +# NO build, so the authenticated Gitea feed is never needed at image-build time. Regenerate +# ./publish with: +# dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \ +# -c Release -o docker/publish -p:UseAppHost=false +# +# The published IL is architecture-neutral, so the multi-arch aspnet:10.0 base runs as-is on +# both Apple Silicon and amd64 hosts. +FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime +WORKDIR /app + +# The base image declares its own binding — as ASPNETCORE_HTTP_PORTS=8080 on .NET 8+, NOT the +# ASPNETCORE_URLS everyone reaches for first (clearing only URLS leaves the warning in place, which +# is a good way to spend an afternoon). Left set, it collides with the Kestrel endpoint section in +# appsettings.Docker.json: Kestrel wins, but logs "Overriding address(es) 'http://*:8080'" on every +# boot — the family's URLs-override trap arriving by default rather than by mistake. Clearing both +# at the layer that introduced them leaves the Kestrel section as the single authority. +ENV ASPNETCORE_URLS= \ + ASPNETCORE_HTTP_PORTS= + +COPY publish/ ./ + +# Dashboard + /health/* + /metrics, all HTTP/1.1 and all anonymous. +EXPOSE 5320 + +ENTRYPOINT ["dotnet", "ZB.MOM.WW.Overview.dll"] diff --git a/ZB.MOM.WW.Overview/docker/README.md b/ZB.MOM.WW.Overview/docker/README.md new file mode 100644 index 0000000..5d6fbee --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/README.md @@ -0,0 +1,81 @@ +# `docker/` — containerised overview dashboard + +Runtime-only image, the HistorianGateway pattern: publish on the host (which has the Gitea feed +credentials), copy the output into `mcr.microsoft.com/dotnet/aspnet:10.0`. Docker never restores. + +```bash +# from ZB.MOM.WW.Overview/ +dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \ + -c Release -o docker/publish -p:UseAppHost=false + +cd docker && docker compose up -d --build +open http://localhost:5320 +``` + +`docker/publish/` is build output and is gitignored. + +## Why the container, and not just `dotnet run` + +The dev rigs publish only some node ports to the host: + +| Fleet | Health endpoint | Reachable from the Mac host? | +|---|---|---| +| ScadaBridge central-a/b | container `:5000` | yes — `localhost:9001` / `:9002` | +| ScadaBridge site-\* (6) | container `:8084` | **no** — not published | +| OtOpcUa central-1/2 | container `:9000` | **no** — only Traefik `:9200`, load-balanced across the pair | +| OtOpcUa site-\* (4) | container `:8080` | **no** — not published | +| HistorianGateway | container `:5220` | yes — `localhost:5220` | +| MxAccessGateway | windev `:5130` | yes, with the VPN up | + +So a host-run dashboard can only see part of the fleet. This stack joins the three rig networks +(`otopcua-dev_default`, `scadabridge-net`, `zb-historiangw_default`, all declared `external`) and +addresses every node by container DNS name, which is how it sees all of them. + +Ports above were established by probing the running rigs on 2026-07-24, **not** read off the +compose files — the two disagree. In particular OtOpcUa's site nodes serve health on `:8080`, not +the `:9000` their compose anchor implies, because their explicit Kestrel listeners for LocalDb sync +re-bind the primary HTTP port. + +Start the rigs before this stack. The networks are `external`, so bringing the dashboard up or +down never disturbs a running rig. + +## Registry shape + +Bound from the `Overview` section; `appsettings.Docker.json` is the worked example. + +```jsonc +"Overview": { + "PollIntervalSeconds": 10, // sweep cadence; per-app and per-instance overrides allowed + "TimeoutSeconds": 3, // per-request timeout + "StaleAfterSeconds": 45, // must exceed the poll interval, or every card reads Stale + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", // label on each instance's management link + "Instances": [ + { + "Name": "central-a", + "Group": "central", // one Akka cluster = one group; leader + split-brain + // detection are scoped to it. Omit for standalone apps. + "BaseUrl": "http://host:5000", // /health/ready and /health/active are probed under this + "ManagementUrl": "http://localhost:9001/", // followed by the OPERATOR'S browser, so it + // must be host-reachable, not container DNS + "HasActiveRole": true // false for single-instance gateways: no Active badge, + // and /health/active is never probed + } + ] + } + ] +} +``` + +`appsettings.json` ships the tuning defaults with an **empty** `Applications` list, so a deployment +that supplies no registry fails at startup naming the missing key rather than booting into an empty +dashboard. + +## Configuration traps + +- **Never set `ASPNETCORE_URLS` alongside `Kestrel__Endpoints__*`.** Declaring both puts Kestrel + into explicit-endpoints mode and one of them is silently discarded. +- No `env_file`, no secrets, no login. The dashboard reads anonymous health endpoints and holds no + credentials. That is a requirement of the design, not an omission. diff --git a/ZB.MOM.WW.Overview/docker/docker-compose.yml b/ZB.MOM.WW.Overview/docker/docker-compose.yml new file mode 100644 index 0000000..755307b --- /dev/null +++ b/ZB.MOM.WW.Overview/docker/docker-compose.yml @@ -0,0 +1,54 @@ +# Local Docker deployment of the SCADA family overview dashboard. +# +# Why this exists rather than just `dotnet run`: the dev rigs publish only some node ports to the +# host. OtOpcUa publishes no per-node HTTP port at all (only Traefik's load-balanced :9200, which +# round-robins the central pair and so cannot identify a node), and ScadaBridge's site nodes keep +# their health port container-internal. A dashboard on the host can therefore see a fraction of the +# fleet; one joined to the rig networks sees all of it, addressing each node by container DNS name. +# +# Registry: appsettings.Docker.json, baked into the image (ASPNETCORE_ENVIRONMENT=Docker below). +# +# No env_file and no secrets: the dashboard reads anonymous /health endpoints, holds no +# credentials, and serves no login. That is a requirement, not an oversight. +# +# Explicit project name so this stack is isolated from the other compose projects on the host. +name: zb-overview + +services: + overview: + build: + context: . + dockerfile: Dockerfile + image: zb-overview:local + container_name: zb-overview + restart: unless-stopped + ports: + - "5320:5320" # dashboard + /health/* + /metrics + environment: + ASPNETCORE_ENVIRONMENT: Docker + + # The endpoint lives in appsettings.Docker.json — ONE authority for the binding. Do NOT add + # ASPNETCORE_URLS here; the Dockerfile clears the base image's default for that reason. + + # No container healthcheck on purpose. aspnet:10.0 ships neither curl nor wget, and adding a + # `--healthcheck` mode to the app to work around that would put a second entry point into + # production code purely to satisfy Compose. The app's real health surface is HTTP and already + # honest — `curl localhost:5320/health/ready` from the host, or scrape /metrics. + + networks: + - otopcua + - scadabridge + - historiangw + +# All three are created by the rigs themselves; this stack joins them and never defines them, so +# bringing the dashboard up or down cannot disturb a running rig. Start the rigs first. +networks: + otopcua: + name: otopcua-dev_default + external: true + scadabridge: + name: scadabridge-net + external: true + historiangw: + name: zb-historiangw_default + external: true diff --git a/ZB.MOM.WW.Overview/nuget.config b/ZB.MOM.WW.Overview/nuget.config new file mode 100644 index 0000000..88c7e71 --- /dev/null +++ b/ZB.MOM.WW.Overview/nuget.config @@ -0,0 +1,24 @@ + + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/App.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/App.razor new file mode 100644 index 0000000..eabf1ce --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/App.razor @@ -0,0 +1,22 @@ +@* Host page. Head order matters: Bootstrap first (the Theme kit's shell and + TechButton are built on Bootstrap utility classes), then which + brings the design tokens and IBM Plex, then site.css so this app's own + dashboard anatomy can override either. *@ + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Layout/MainLayout.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Layout/MainLayout.razor new file mode 100644 index 0000000..9203912 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Layout/MainLayout.razor @@ -0,0 +1,43 @@ +@inherits LayoutComponentBase +@inject OverviewSnapshotStore Store + +@* Thin layout: the side-rail chassis belongs to the shared kit's . + No AuthorizeView anywhere — this dashboard is anonymous by requirement, so + there is no user to show and nothing to sign out of. The rail's application + links are fragment anchors into the single page rather than routes: the whole + fleet is one view, and splitting it per application would defeat the point. *@ + + + +
read-only · anonymous
ZB.MOM.WW.Overview @Version
+ + @Body + + +@code { + private static readonly string Version = + typeof(MainLayout).Assembly.GetName().Version?.ToString(3) ?? "0.0.0"; + + private IReadOnlyList Applications => + Store.Current.Applications.Select(a => a.Name).ToList(); + + /// + /// Builds the fragment id for an application section. Must stay identical to the id the + /// Overview page emits, or every rail link becomes a dead anchor. + /// + /// The application's display name. + /// A URL-fragment-safe slug. + internal static string Slug(string name) => + new(name.Where(c => char.IsLetterOrDigit(c) || c is '-' or '_').Select(char.ToLowerInvariant).ToArray()); +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Pages/Overview.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Pages/Overview.razor new file mode 100644 index 0000000..6d08ee9 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Pages/Overview.razor @@ -0,0 +1,245 @@ +@page "/" +@implements IDisposable +@inject OverviewSnapshotStore Store +@inject TimeProvider Time + +@* The whole dashboard, rendered from the snapshot store. This page never polls + an instance: it subscribes to the store and re-renders when the poller swaps + a snapshot in, plus a slow local tick so ages and staleness advance on their + own between sweeps. *@ + +@* Read once per render and threaded through everything below, so every card, KPI + and group header on one paint agrees on what "now" is. Reading the clock at + render — rather than caching it when a snapshot arrives — is what lets the view + age on its own: a paused or stuck poller publishes nothing, and the cards still + cross into Stale. *@ +@{ + var now = Time.GetUtcNow(); +} + +SCADA Overview + +
+

Fleet Overview

+ @Meta + + + @(_paused ? "Paused" : "Polling") + + +
+ +@if (_snapshot.Applications.Count == 0) +{ +

+ No instances have been polled yet. +

+} +else +{ +
+
+
Instances
+
@_instances.Countregistered
+
+
+
Up
+
@Count(now, InstanceStatus.Up)
+
+
+
Degraded
+
@Count(now, InstanceStatus.Degraded)
+
+
+
Down
+
@Count(now, InstanceStatus.Down)
+
+
+
Unreachable
+
@Count(now, InstanceStatus.Unreachable)
+
+
+
Stale
+
@StaleCount(now)
+
+
+ + @foreach (var application in _snapshot.Applications) + { +
+
+

@application.Name

+ @RollupLabel(application) + + @AppMeta(application) +
+ + @foreach (var group in Layout(application)) + { +
+ @if (group.Key is { } groupName) + { + var state = application.GroupFor(groupName); +
+ @groupName + @if (state?.LeaderInstance is { } leaderInstance) + { + Leader · @leaderInstance + } + else if (state?.LeaderAddress is { } leaderAddress) + { + @* Unresolved address: shown raw rather than hidden — a leader on a + host that is not in the registry is itself the finding. *@ + Leader · @leaderAddress + } + @if (state?.Disagreement == true) + { + + Split brain · @string.Join(" vs ", state.ReportedLeaders) + + } + @GroupMeta(group, now) +
+ } +
+ @foreach (var instance in group) + { + + } +
+
+ } +
+ } +} + +@code { + /// + /// How often the page re-renders on its own. Independent of the poll interval: its job is to + /// age the "last poll" figures and let a card cross into Stale even when no sweep lands — + /// which is exactly the case a stuck poller produces. + /// + private static readonly TimeSpan UiTick = TimeSpan.FromSeconds(5); + + private OverviewSnapshot _snapshot = OverviewSnapshot.Empty; + private IReadOnlyList _instances = []; + private bool _paused; + private ITimer? _timer; + + protected override void OnInitialized() + { + Adopt(Store.Current); + Store.Changed += OnSnapshotChanged; + _timer = Time.CreateTimer(_ => InvokeAsync(StateHasChanged), null, UiTick, UiTick); + } + + private void OnSnapshotChanged(OverviewSnapshot snapshot) + { + // Pause stops ADOPTING new snapshots; the poller keeps sweeping. That is what makes the + // pause useful — the frozen view then visibly ages into Stale, so a paused dashboard can + // never be mistaken for a live one. + if (_paused) + return; + + InvokeAsync(() => + { + Adopt(snapshot); + StateHasChanged(); + }); + } + + private void Adopt(OverviewSnapshot snapshot) + { + _snapshot = snapshot; + _instances = snapshot.AllInstances.ToList(); + } + + private void TogglePause() + { + _paused = !_paused; + + if (!_paused) + Adopt(Store.Current); + } + + private int Count(DateTimeOffset now, InstanceStatus status) => + _instances.Count(i => i.Status == status && !i.IsStale(now)); + + private int StaleCount(DateTimeOffset now) => _instances.Count(i => i.IsStale(now)); + + private string Meta + { + get + { + var generated = _snapshot.GeneratedUtc == default + ? "never" + : _snapshot.GeneratedUtc.ToLocalTime().ToString("HH:mm:ss", System.Globalization.CultureInfo.InvariantCulture); + + return $"last sweep {generated} · {_instances.Count} instances"; + } + } + + /// + /// Groups an application's instances for display: named cluster groups first in registry + /// order, then the ungrouped ones. + /// + /// The application to lay out. + /// The instance groups. + private static IEnumerable> Layout(ApplicationSnapshot application) => + application.Instances + .GroupBy(i => i.Group, StringComparer.Ordinal) + .OrderBy(g => g.Key is null) + .ThenBy(g => g.Key, StringComparer.Ordinal); + + private static string AppMeta(ApplicationSnapshot application) + { + var clusters = application.Groups.Count; + var instances = application.Instances.Count; + var clusterText = clusters == 0 ? "" : $" · {clusters} cluster{(clusters == 1 ? "" : "s")}"; + + return $"{instances} instance{(instances == 1 ? "" : "s")}{clusterText}"; + } + + private static string GroupMeta(IGrouping group, DateTimeOffset now) + { + var members = group.Count(); + var missing = group.Count(i => i.Status is InstanceStatus.Unreachable || i.IsStale(now)); + + return $"{members} member{(members == 1 ? "" : "s")} · {missing} unreachable"; + } + + private static bool IsLeader(ApplicationSnapshot application, InstanceSnapshot instance) => + instance.Group is { } group && + application.GroupFor(group)?.LeaderInstance is { } leader && + string.Equals(leader, instance.Instance, StringComparison.Ordinal); + + private static StatusState RollupState(ApplicationSnapshot application) => application.Worst switch + { + InstanceStatus.Up => StatusState.Ok, + InstanceStatus.Degraded => StatusState.Warn, + InstanceStatus.Down or InstanceStatus.Unreachable => StatusState.Bad, + _ => StatusState.Idle, + }; + + private static string RollupLabel(ApplicationSnapshot application) => application.Worst switch + { + InstanceStatus.Up => "Up", + InstanceStatus.Degraded => "Degraded", + InstanceStatus.Down => "Down", + InstanceStatus.Unreachable => "Unreachable", + _ => "Pending", + }; + + /// + public void Dispose() + { + // Both are required: an un-detached handler would call StateHasChanged on a disposed + // renderer once per sweep for the life of the process. + Store.Changed -= OnSnapshotChanged; + _timer?.Dispose(); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Routes.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Routes.razor new file mode 100644 index 0000000..df6e20b --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Routes.razor @@ -0,0 +1,18 @@ +@* Plain RouteView, not AuthorizeRouteView. The dashboard is anonymous and + read-only by requirement (design §2): there is no authentication pipeline to + authorize against, and wiring one in would be the opposite of the decision. *@ + + + + + + + + Overview — Not Found +
+

Not Found

+

The requested page does not exist.

+
+ + + diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Widgets/InstanceCard.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Widgets/InstanceCard.razor new file mode 100644 index 0000000..2ef9ce7 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Widgets/InstanceCard.razor @@ -0,0 +1,300 @@ +@* One instance = one card. Every value shown here comes from the snapshot the + poller published; the component never probes anything itself. *@ + +
+
+
+ @Instance.Instance + @HostAndPort +
+
+ @if (IsLeader) + { + Leader + } + @if (Instance.Active is ActiveState.Active) + { + Active + } + else if (Instance.Active is ActiveState.Standby) + { + Standby + } + else if (Instance.Active is ActiveState.Unknown) + { + Role ? + } + @StatusLabel +
+
+ +
+
+ Latency + @LatencyText +
+
+ @(IsStale ? "Stale since" : "Last poll") + @LastPollText +
+
+ Checks + @ChecksText +
+ + @if (Instance.Error is { Length: > 0 } error) + { +
+ Error + @Truncate(error, 48) +
+ } + + @if (Entries.Count > 0) + { +
+ @Entries.Count check@(Entries.Count == 1 ? "" : "s") + @foreach (var entry in Entries) + { +
+ + @entry.Key + @entry.Value.Description +
+ } + @if (ClusterDataLine is { Length: > 0 } clusterData) + { + @* The cluster view, rendered verbatim rather than interpreted: when a pair + disagrees about the leader this line is the raw evidence an operator needs + to see, not a summary of it. *@ +
@((MarkupString)clusterData)
+ } +
+ } +
+ +
+ @if (Instance.ManagementUrl is { Length: > 0 } managementUrl) + { + Open @Instance.ManagementLabel ↗ + } + else + { + no management link + } + @RawSignals +
+
+ +@code { + /// The instance to render. + [Parameter, EditorRequired] public InstanceSnapshot Instance { get; set; } = default!; + + /// The render-time clock, passed down so every card on a page agrees on "now". + [Parameter, EditorRequired] public DateTimeOffset Now { get; set; } + + /// Whether this instance is its group's leader. + [Parameter] public bool IsLeader { get; set; } + + private bool IsStale => Instance.IsStale(Now); + + private IReadOnlyList> Entries => + Instance.Report?.Entries.OrderBy(e => e.Key, StringComparer.Ordinal).ToList() ?? []; + + /// + /// Stale wins over the stored status in the class, so the card visibly greys out. What it + /// shows is still the last real reading — the point is to mark it as no longer current, not + /// to replace it with a guess. + /// + private string StateClass => Instance.Status switch + { + null => "pending", + _ when IsStale => $"{Instance.Status.ToString()!.ToLowerInvariant()} stale", + var status => status.ToString()!.ToLowerInvariant(), + }; + + private string StatusLabel => Instance.Status switch + { + null => "Pending", + _ when IsStale => "Stale", + InstanceStatus.Up => "Up", + InstanceStatus.Degraded => "Degraded", + InstanceStatus.Down => "Down", + _ => "Unreachable", + }; + + private StatusState StatusPillState => Instance.Status switch + { + null => StatusState.Idle, + _ when IsStale => StatusState.Idle, + InstanceStatus.Up => StatusState.Ok, + InstanceStatus.Degraded => StatusState.Warn, + _ => StatusState.Bad, + }; + + private string HostAndPort + { + get + { + if (!Uri.TryCreate(Instance.BaseUrl, UriKind.Absolute, out var uri)) + return Instance.BaseUrl; + + return uri.IsDefaultPort ? uri.Host : $"{uri.Host}:{uri.Port}"; + } + } + + private string LatencyText => + Instance.Status is null ? "—" : $"{Instance.Latency.TotalMilliseconds:0} ms"; + + private string LastPollText + { + get + { + // Stale cards count from the last successful contact, not the last poll attempt: + // "stale since 4 s ago" while an instance has actually been gone for ten minutes + // would be actively misleading. + var reference = IsStale ? Instance.LastReachableUtc ?? Instance.LastPolledUtc : Instance.LastPolledUtc; + return reference is { } at ? Relative(Now - at) : "never"; + } + } + + private string ChecksText + { + get + { + if (Entries.Count == 0) + return Instance.Status is null ? "—" : "no detail"; + + var healthy = Entries.Count(e => string.Equals(e.Value.Status, "Healthy", StringComparison.Ordinal)); + return $"{healthy} / {Entries.Count} healthy"; + } + } + + private string? ChecksClass + { + get + { + if (Entries.Count == 0) + return null; + + if (Entries.Any(e => string.Equals(e.Value.Status, "Unhealthy", StringComparison.Ordinal))) + return "bad"; + + return Entries.Any(e => string.Equals(e.Value.Status, "Degraded", StringComparison.Ordinal)) + ? "warn" + : "ok"; + } + } + + private string RawSignals + { + get + { + var ready = Instance.Status switch + { + null => "—", + InstanceStatus.Unreachable => "fail", + InstanceStatus.Down => "503", + _ => "200", + }; + + var active = Instance.Active switch + { + ActiveState.Active => "200", + ActiveState.Standby => "503", + ActiveState.Unknown => "—", + _ => null, + }; + + return active is null ? $"ready {ready}" : $"ready {ready} · active {active}"; + } + } + + /// + /// Renders the cluster check's data as the mockup's monospace evidence line. + /// + /// + /// + /// Located by the data it carries rather than by check name: the two apps register the + /// same shared checks under different names (akka-cluster vs akka, + /// active-node vs cluster-primary), so keying on either one silently drops + /// the evidence line for the other. + /// + /// + /// active and leader are shown side by side on purpose. They come + /// from different checks and are different nodes whenever any node has restarted — the + /// leader is the lowest-addressed Up member, the active node is the oldest. Showing both + /// turns the confusion that produced lmxopcua#494 into something an operator can read + /// straight off the card, instead of a silent disagreement between the chip and the tier. + /// + /// + private string? ClusterDataLine + { + get + { + var entries = Instance.Report?.Entries.Values; + if (entries is null) + return null; + + var cluster = entries.FirstOrDefault(e => e.DataString(LeaderResolver.LeaderDataKey) is { Length: > 0 }); + var active = entries.FirstOrDefault(e => e.DataString(LeaderResolver.ActiveNodeDataKey) is { Length: > 0 }); + + if (cluster is null && active is null) + return null; + + var parts = new List(); + + void Add(string label, string? value) + { + if (!string.IsNullOrEmpty(value)) + parts.Add($"{Escape(label)} {Escape(value)}"); + } + + Add("active", active?.DataString(LeaderResolver.ActiveNodeDataKey)); + Add("leader", cluster?.DataString(LeaderResolver.LeaderDataKey)); + Add("members", cluster?.DataInt("memberCount")?.ToString()); + Add("unreachable", cluster?.DataInt("unreachableCount")?.ToString()); + Add("self", (cluster ?? active)?.DataString("selfAddress")); + + return parts.Count == 0 ? null : string.Join(" · ", parts); + } + } + + private static string DotClass(string? status) => status switch + { + "Healthy" => "ok", + "Degraded" => "warn", + "Unhealthy" => "bad", + _ => "idle", + }; + + /// Formats an age the way an operator reads a dashboard: coarse and instant. + /// The elapsed time. + /// A short relative string. + internal static string Relative(TimeSpan age) + { + if (age < TimeSpan.Zero) + age = TimeSpan.Zero; + + if (age.TotalSeconds < 60) + return $"{age.TotalSeconds:0} s ago"; + + if (age.TotalMinutes < 60) + return $"{age.TotalMinutes:0} min ago"; + + return age.TotalHours < 24 ? $"{age.TotalHours:0} h ago" : $"{age.TotalDays:0} d ago"; + } + + private static string Truncate(string value, int max) => + value.Length <= max ? value : string.Concat(value.AsSpan(0, max - 1), "…"); + + /// + /// HTML-escapes a value before it goes into the one MarkupString on this card. Health data is + /// remote input from another application — trusted operationally, but never trusted to be + /// markup-safe. + /// + /// The raw value. + /// The escaped value. + private static string Escape(string value) => + System.Net.WebUtility.HtmlEncode(value); +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/_Imports.razor b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/_Imports.razor new file mode 100644 index 0000000..61d86d3 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/_Imports.razor @@ -0,0 +1,10 @@ +@using Microsoft.AspNetCore.Components.Routing +@using Microsoft.AspNetCore.Components.Web +@using ZB.MOM.WW.Overview.Components +@using ZB.MOM.WW.Overview.Components.Layout +@using ZB.MOM.WW.Overview.Components.Pages +@using ZB.MOM.WW.Overview.Components.Widgets +@using ZB.MOM.WW.Overview.Polling +@using ZB.MOM.WW.Overview.Registry +@using ZB.MOM.WW.Theme +@using static Microsoft.AspNetCore.Components.Web.RenderMode diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs new file mode 100644 index 0000000..37fdb57 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/ObservabilityServiceCollectionExtensions.cs @@ -0,0 +1,37 @@ +using ZB.MOM.WW.Health; + +namespace ZB.MOM.WW.Overview.Observability; + +/// Registers the dashboard's own health checks. +public static class ObservabilityServiceCollectionExtensions +{ + /// Check name for the registry-loaded probe. + public const string RegistryLoadedCheckName = "registry-loaded"; + + /// Check name for the poll-cycle liveness probe. + public const string PollCycleCheckName = "last-poll-cycle-completed"; + + /// + /// Adds the two readiness checks that make the dashboard's own /health/ready mean + /// something: that it bound a registry, and that it is still polling it. + /// + /// The service collection. + /// The same collection, for chaining. + /// + /// Both are tagged only. The dashboard is a single stateless + /// process with no active/standby concept, so it registers nothing on the active tier — its own + /// /health/active is deliberately an empty, always-200 report. + /// + public static IServiceCollection AddOverviewHealthChecks(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.AddHealthChecks() + .AddTypeActivatedCheck( + RegistryLoadedCheckName, failureStatus: null, tags: [ZbHealthTags.Ready]) + .AddTypeActivatedCheck( + PollCycleCheckName, failureStatus: null, tags: [ZbHealthTags.Ready]); + + return services; + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs new file mode 100644 index 0000000..dfc67e3 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/OverviewMetrics.cs @@ -0,0 +1,90 @@ +using System.Diagnostics.Metrics; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// The dashboard's own Meter: what it currently believes about every registered instance, and how +/// long its sweeps take. +/// +/// +/// +/// The instance gauge is observable — it is read from the published snapshot at scrape +/// time rather than written on every poll. The snapshot is already the single authority for what +/// the dashboard believes, so deriving the gauge from it makes a disagreement between the page and +/// the metric impossible by construction. +/// +/// +/// The instrument names follow the family convention in +/// components/observability/spec/METRIC-CONVENTIONS.md §2 — lower-case, dot-separated +/// <app>.<subsystem>.<event>, durations in seconds — so the Prometheus +/// exporter renders them as overview_instance_status and +/// overview_poll_duration_seconds. +/// +/// +public sealed class OverviewMetrics : IDisposable +{ + /// The app's single Meter name, per the family one-meter-per-app convention. + public const string MeterName = "ZB.MOM.WW.Overview"; + + /// Per-instance status gauge; the value is the ordinal. + public const string InstanceStatusInstrument = "overview.instance.status"; + + /// Sweep-duration histogram, in seconds. + public const string PollDurationInstrument = "overview.poll.duration"; + + private readonly Meter _meter; + private readonly Histogram _pollDuration; + private readonly OverviewSnapshotStore _store; + + /// Initializes the meter and its instruments. + /// Factory that scopes the meter's lifetime to the container. + /// The snapshot the observable gauge reads at scrape time. + public OverviewMetrics(IMeterFactory meterFactory, OverviewSnapshotStore store) + { + ArgumentNullException.ThrowIfNull(meterFactory); + + _store = store ?? throw new ArgumentNullException(nameof(store)); + _meter = meterFactory.Create(MeterName); + + _meter.CreateObservableGauge( + InstanceStatusInstrument, + ObserveInstanceStatus, + unit: "1", + description: "Derived status of each registered instance (0=Up, 1=Degraded, 2=Down, 3=Unreachable)."); + + _pollDuration = _meter.CreateHistogram( + PollDurationInstrument, + unit: "s", + description: "Wall-clock duration of one poll sweep across every due instance."); + } + + /// Records the duration of one completed sweep. + /// Wall-clock time the sweep took. + public void RecordSweep(TimeSpan duration) => _pollDuration.Record(duration.TotalSeconds); + + /// + public void Dispose() => _meter.Dispose(); + + private IEnumerable> ObserveInstanceStatus() + { + foreach (var instance in _store.Current.AllInstances) + { + // Instances that have never been probed are skipped rather than reported as anything. + // Emitting a value for them would publish a status the dashboard does not yet hold — + // and the enum's zero ordinal is Up, so the fabrication would be the reassuring one. + if (instance.Status is not { } status) + continue; + + yield return new Measurement( + (int)status, + new KeyValuePair("application", instance.Application), + + // Deliberately NOT named "instance": Prometheus attaches its own `instance` label to + // every scraped series, and a colliding metric label is silently renamed to + // `exported_instance`, which no dashboard query would be written against. + new KeyValuePair("node", instance.Instance), + new KeyValuePair("group", instance.Group ?? string.Empty)); + } + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs new file mode 100644 index 0000000..4aeb8c9 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHealthCheck.cs @@ -0,0 +1,110 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// Fails when the poller has stopped sweeping — the one failure mode that would otherwise leave the +/// dashboard looking perfectly healthy while every card it serves quietly went stale. +/// +/// +/// This is what makes the dashboard's own /health/ready honest. Without it the process +/// answers 200 for as long as Kestrel is up, which says nothing about whether the data behind the +/// page is still being refreshed. +/// +public sealed class PollCycleHealthCheck : IHealthCheck +{ + private readonly PollCycleHeartbeat _heartbeat; + private readonly TimeProvider _timeProvider; + private readonly TimeSpan _grace; + + /// Initializes a new . + /// The poller's completed-sweep heartbeat. + /// The registry, used to derive the tolerated gap. + /// Clock. + public PollCycleHealthCheck( + PollCycleHeartbeat heartbeat, + IOptions options, + TimeProvider timeProvider) + { + ArgumentNullException.ThrowIfNull(options); + + _heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _grace = GraceFor(options.Value); + } + + /// + /// The longest gap between completed sweeps that is still considered healthy. + /// + /// The registry. + /// The tolerated gap. + /// + /// A sweep completes on every timer tick, and the timer ticks at the fastest configured + /// interval — so the expected gap is one interval plus however long the probes took, which is + /// bounded by the slowest configured timeout. Twice that is comfortably past "slow" and + /// unambiguously "stopped". + /// + internal static TimeSpan GraceFor(OverviewOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var timings = options.Applications + .SelectMany(app => app.Instances.Select(instance => + EffectiveTimings.Resolve(options, app, instance))) + .ToList(); + + // An empty registry cannot poll at all; RegistryLoadedHealthCheck is the check that reports + // that, so fall back to the global defaults rather than dividing by an empty set here. + var interval = timings.Count > 0 + ? timings.Min(t => t.PollInterval) + : TimeSpan.FromSeconds(options.PollIntervalSeconds); + var timeout = timings.Count > 0 + ? timings.Max(t => t.Timeout) + : TimeSpan.FromSeconds(options.TimeoutSeconds); + + return 2 * (interval + timeout); + } + + /// + public Task CheckHealthAsync( + HealthCheckContext context, + CancellationToken cancellationToken = default) + { + var now = _timeProvider.GetUtcNow(); + var last = _heartbeat.LastCompletedUtc; + + var data = new Dictionary(StringComparer.Ordinal) + { + ["graceSeconds"] = Math.Round(_grace.TotalSeconds, 3), + }; + + if (last is null) + { + // Nothing has swept yet. Within the grace window that is ordinary startup, so report + // Degraded (still a 200) rather than failing readiness for the first few seconds of + // every deployment. Past it, the poller never started and that IS a failure. + var waiting = now - _heartbeat.StartedUtc; + data["waitingSeconds"] = Math.Round(waiting.TotalSeconds, 3); + + return Task.FromResult(waiting <= _grace + ? HealthCheckResult.Degraded("No poll cycle has completed yet; the poller is still starting.", data: data) + : HealthCheckResult.Unhealthy( + $"No poll cycle has completed in {waiting.TotalSeconds:0.#}s since startup.", + exception: null, + data)); + } + + var age = now - last.Value; + data["lastCycleUtc"] = last.Value.ToString("O", System.Globalization.CultureInfo.InvariantCulture); + data["ageSeconds"] = Math.Round(age.TotalSeconds, 3); + + return Task.FromResult(age <= _grace + ? HealthCheckResult.Healthy($"Last poll cycle completed {age.TotalSeconds:0.#}s ago.", data) + : HealthCheckResult.Unhealthy( + $"No poll cycle has completed in {age.TotalSeconds:0.#}s; the displayed data is stale.", + exception: null, + data)); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs new file mode 100644 index 0000000..5f6de79 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/PollCycleHeartbeat.cs @@ -0,0 +1,46 @@ +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// When the poller last finished a full sweep. Written by the poller, read by +/// . +/// +/// +/// Deliberately not derived from the published snapshot's GeneratedUtc: the poller publishes +/// the registry once before it probes anything, so a snapshot exists from the moment the host +/// starts. A stuck poller would therefore look like a working one. This records only sweeps that +/// actually completed. +/// +public sealed class PollCycleHeartbeat +{ + private long _lastCompletedTicks; + + /// Initializes the heartbeat and stamps the host start time. + /// Clock. + public PollCycleHeartbeat(TimeProvider timeProvider) + { + ArgumentNullException.ThrowIfNull(timeProvider); + StartedUtc = timeProvider.GetUtcNow(); + } + + /// When this heartbeat was constructed — effectively when the host started. + /// + /// Lets the health check distinguish "started a moment ago and has not swept yet" from "has + /// not swept in minutes", which are the same null reading but opposite conclusions. + /// + public DateTimeOffset StartedUtc { get; } + + /// When the last full sweep completed, or null if none has. + public DateTimeOffset? LastCompletedUtc + { + get + { + var ticks = Interlocked.Read(ref _lastCompletedTicks); + return ticks == 0 ? null : new DateTimeOffset(ticks, TimeSpan.Zero); + } + } + + /// Records that a sweep completed. + /// When it completed. + public void RecordCycleCompleted(DateTimeOffset at) => + Interlocked.Exchange(ref _lastCompletedTicks, at.UtcTicks); +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs new file mode 100644 index 0000000..92c9ba7 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Observability/RegistryLoadedHealthCheck.cs @@ -0,0 +1,52 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Observability; + +/// +/// Reports how much registry the dashboard actually bound, and fails if it bound none. +/// +/// +/// The pre-host ConfigPreflight and the options validator already refuse to start on an +/// empty registry, so the unhealthy branch is a backstop rather than an expected state. The check +/// earns its place on the reported counts: an operator looking at /health/ready after a +/// config change can see how many instances this process is watching without reading its +/// configuration files. +/// +public sealed class RegistryLoadedHealthCheck : IHealthCheck +{ + private readonly OverviewOptions _options; + + /// Initializes a new . + /// The bound registry. + public RegistryLoadedHealthCheck(IOptions options) + { + ArgumentNullException.ThrowIfNull(options); + _options = options.Value; + } + + /// + public Task CheckHealthAsync( + HealthCheckContext context, + CancellationToken cancellationToken = default) + { + var applications = _options.Applications.Count; + var instances = _options.Applications.Sum(a => a.Instances.Count); + + var data = new Dictionary(StringComparer.Ordinal) + { + ["applications"] = applications, + ["instances"] = instances, + }; + + return Task.FromResult(instances > 0 + ? HealthCheckResult.Healthy( + $"Watching {instances} instance(s) across {applications} application(s).", + data) + : HealthCheckResult.Unhealthy( + "The registry contains no instances; the dashboard has nothing to watch.", + exception: null, + data)); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/InstanceStatus.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/InstanceStatus.cs new file mode 100644 index 0000000..90b011c --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/InstanceStatus.cs @@ -0,0 +1,132 @@ +namespace ZB.MOM.WW.Overview.Polling; + +/// The derived state of one probed instance. +public enum InstanceStatus +{ + /// Ready tier answered 200 / Healthy. + Up, + + /// Ready tier answered 200 / Degraded — serving, but a check is unhappy. + Degraded, + + /// + /// Ready tier answered 503 / Unhealthy. The app is up and talking; a dependency is not. + /// + Down, + + /// + /// Nothing answered — timeout, connection refused, DNS failure, or a body that is not the + /// canonical health JSON. Kept distinct from because it usually means + /// network / VPN / registry typo rather than a sick application, and the two lead an operator + /// to look in completely different places. + /// + Unreachable, +} + +/// Whether a probed instance is the active half of its pair. +public enum ActiveState +{ + /// The instance has no active/standby concept (single-instance gateways). + NotApplicable, + + /// The active probe did not answer, so activity is unknown. + Unknown, + + /// Active tier answered 200 — this is the active node. + Active, + + /// Active tier answered 503 — this is the standby. + Standby, +} + +/// +/// Pure status derivation: probe result → , with two-strike flap +/// damping applied across polls. Static and dependency-free so every rule is table-testable. +/// +public static class StatusDerivation +{ + /// + /// Consecutive failing observations required before a previously-good instance is shown as + /// failing. A single blip — one dropped packet, one GC pause past the timeout — is not news; + /// two in a row is. + /// + public const int FailureStrikes = 2; + + /// Classifies one ready-tier probe, before any damping. + /// The probe result. + /// The observed status. + public static InstanceStatus Observe(HealthProbeResult probe) + { + ArgumentNullException.ThrowIfNull(probe); + + if (!probe.Reachable || probe.Report is null) + return InstanceStatus.Unreachable; + + // The reported status wins over the HTTP code: they agree by construction (Healthy/Degraded + // → 200, Unhealthy → 503), and the body is the more specific of the two — 200 alone cannot + // tell Up from Degraded. + return probe.Report.Status switch + { + "Healthy" => InstanceStatus.Up, + "Degraded" => InstanceStatus.Degraded, + "Unhealthy" => InstanceStatus.Down, + + // Parsed as JSON but carrying no status we recognise: the endpoint is answering with + // something that is not this contract, which is the Unreachable case, not a sick app. + _ => InstanceStatus.Unreachable, + }; + } + + /// Classifies one active-tier probe. + /// Whether this instance participates in an active/standby pair. + /// The active-tier probe result, or null when not probed. + /// The derived active state. + public static ActiveState ObserveActive(bool hasActiveRole, HealthProbeResult? probe) + { + if (!hasActiveRole) + return ActiveState.NotApplicable; + + if (probe is null || !probe.Reachable || probe.StatusCode is null) + return ActiveState.Unknown; + + return probe.StatusCode switch + { + 200 => ActiveState.Active, + 503 => ActiveState.Standby, + _ => ActiveState.Unknown, + }; + } + + /// + /// Applies two-strike damping: a failing observation only replaces a previously-good state + /// after consecutive failures, while any good observation is + /// adopted immediately. + /// + /// The last displayed status, or null on the first-ever poll. + /// Consecutive failures recorded before this poll. + /// This poll's observed status. + /// The status to display and the updated consecutive-failure count. + /// + /// Recovery is deliberately asymmetric — instant. Damping exists to suppress false alarms, and + /// making recovery equally slow would just convert them into false alarms that linger. + /// + public static (InstanceStatus Status, int ConsecutiveFailures) Damp( + InstanceStatus? previous, + int previousConsecutiveFailures, + InstanceStatus observed) + { + var failing = observed is InstanceStatus.Down or InstanceStatus.Unreachable; + + if (!failing) + return (observed, 0); + + var failures = previousConsecutiveFailures + 1; + + // With no previous state there is nothing to protect and nothing better to show: adopt the + // observation. Claiming Up on a first poll that failed would be a fabrication. + if (previous is null || failures >= FailureStrikes) + return (observed, failures); + + return (previous.Value, failures); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/LeaderResolver.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/LeaderResolver.cs new file mode 100644 index 0000000..79f751a --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/LeaderResolver.cs @@ -0,0 +1,186 @@ +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// Derives per-group leader state from whatever health check a member publishes its cluster view +/// under. +/// +/// +/// Pure and static: given the same instance snapshots it always produces the same answer, which is +/// what makes the split-brain rule — the one output an operator would act on at 3am — exhaustively +/// testable without a cluster. +/// +public static class LeaderResolver +{ + /// + /// The data key holding the address of the node actually in charge — the oldest Up member, + /// published by the active tier's ActiveNodeHealthCheck from ZB.MOM.WW.Health 0.3.0. + /// + public const string ActiveNodeDataKey = "activeNode"; + + /// + /// The data key holding the Akka cluster leader address, published by + /// AkkaClusterHealthCheck. Fallback only — see . + /// + public const string LeaderDataKey = "leader"; + + /// Resolves leader state for every cluster group present in . + /// One application's instances. + /// Group state in first-appearance order; empty when the application has no groups. + public static IReadOnlyList Resolve(IEnumerable instances) + { + ArgumentNullException.ThrowIfNull(instances); + + var all = instances as IReadOnlyList ?? instances.ToList(); + var groups = new List(); + + foreach (var name in all + .Select(i => i.Group) + .Where(g => !string.IsNullOrWhiteSpace(g)) + .Distinct(StringComparer.Ordinal)) + { + var members = all.Where(i => string.Equals(i.Group, name, StringComparison.Ordinal)).ToList(); + + // A group of instances that have no active/standby concept has no leader to report; + // rendering a leader chip for one would invent a distinction that does not exist. + if (!members.Exists(m => m.HasActiveRole)) + continue; + + groups.Add(ResolveGroup(name!, members, all)); + } + + return groups; + } + + private static GroupSnapshot ResolveGroup( + string name, + IReadOnlyList members, + IReadOnlyList allInstances) + { + // Only members that are actually talking get a vote. A Down or Unreachable node's last + // known leader is stale by definition, and counting it would manufacture a disagreement + // every time a node goes away mid-failover — the exact moment the flag must stay honest. + var votes = members + .Where(m => m.Status is InstanceStatus.Up or InstanceStatus.Degraded) + .Select(LeaderReportedBy) + .Where(address => !string.IsNullOrWhiteSpace(address)) + .Select(address => address!) + .ToList(); + + if (votes.Count == 0) + return new GroupSnapshot(name, null, null, Disagreement: false, []); + + var distinct = votes.Distinct(StringComparer.Ordinal).OrderBy(a => a, StringComparer.Ordinal).ToList(); + + // The most-reported address wins the chip, ties broken ordinally so the display is stable + // rather than flickering between two equally-supported claims on consecutive sweeps. + var winner = votes + .GroupBy(a => a, StringComparer.Ordinal) + .OrderByDescending(g => g.Count()) + .ThenBy(g => g.Key, StringComparer.Ordinal) + .First() + .Key; + + return new GroupSnapshot( + name, + ResolveToInstanceName(winner, allInstances), + winner, + Disagreement: distinct.Count > 1, + distinct); + } + + /// Finds the address of the in-charge node one instance reports. + /// The instance whose report to search. + /// The reported address, or null when this instance publishes none. + /// + /// + /// activeNode first, leader only as a fallback. They are different + /// nodes and the chip must show the first. The Akka cluster leader is the + /// lowest-addressed Up member — address order, no relationship to time — while the node + /// that actually holds the singletons and answers 200 on /health/active is the + /// oldest Up member. They agree only until some node restarts, after which the + /// restarted node rejoins youngest but keeps its address; observed live on the rebuilt + /// OtOpcUa rig, where the leader was central-1 while central-2 was the + /// active node. Ranking leader first would put the Active chip on the standby. + /// + /// + /// The fallback still matters: it is what an instance running Health < 0.3.0, or one + /// whose active tier was not probed, publishes. Showing the leader there is better than + /// showing nothing, and it is the pre-0.3.0 behaviour unchanged. + /// + /// + /// Matched on the DATA KEY, not on the check name. The two apps register the same shared + /// checks under different names — ScadaBridge calls its cluster check + /// akka-cluster, OtOpcUa calls it akka; the active check is + /// active-node in one and cluster-primary in the other — and a resolver + /// keyed on any of those names silently renders no chip for the other app, which is + /// exactly how this was missed until a live rig showed OtOpcUa's groups bare while + /// ScadaBridge's were fully populated. The data key is the part of the contract the + /// shared checks actually own. + /// + /// + internal static string? LeaderReportedBy(InstanceSnapshot instance) + { + if (instance.Report?.Entries is not { } entries) + return null; + + string? leaderFallback = null; + + foreach (var entry in entries.Values) + { + if (entry.DataString(ActiveNodeDataKey) is { Length: > 0 } activeNode) + return activeNode; + + leaderFallback ??= entry.DataString(LeaderDataKey) is { Length: > 0 } leader ? leader : null; + } + + return leaderFallback; + } + + /// + /// Maps an Akka address to a registry instance name by host. + /// + /// An address of the form akka.tcp://system@host:port. + /// Candidate instances. + /// The matching instance's name, or null when nothing matches. + /// + /// Host only, deliberately: the Akka remoting port and the HTTP port are different by design, so + /// comparing ports would never match. An unmatched address is not an error — the card falls back + /// to showing the raw address, which is still the useful thing to see. + /// + internal static string? ResolveToInstanceName(string akkaAddress, IReadOnlyList instances) + { + var host = HostOf(akkaAddress); + if (host is null) + return null; + + foreach (var instance in instances) + { + if (Uri.TryCreate(instance.BaseUrl, UriKind.Absolute, out var uri) && + string.Equals(uri.Host, host, StringComparison.OrdinalIgnoreCase)) + { + return instance.Instance; + } + } + + return null; + } + + /// Extracts the host from an Akka address. + /// The address to parse. + /// The host, or null when the address is not in the expected form. + internal static string? HostOf(string? akkaAddress) + { + if (string.IsNullOrWhiteSpace(akkaAddress)) + return null; + + var at = akkaAddress.LastIndexOf('@'); + if (at < 0 || at == akkaAddress.Length - 1) + return null; + + var hostAndPort = akkaAddress[(at + 1)..]; + var colon = hostAndPort.LastIndexOf(':'); + var host = colon > 0 ? hostAndPort[..colon] : hostAndPort; + + return host.Length == 0 ? null : host; + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs new file mode 100644 index 0000000..7805f2a --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewPollerService.cs @@ -0,0 +1,255 @@ +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// The single writer: sweeps every registered instance on a timer and publishes each result set to +/// . +/// +/// +/// No retries. A failed probe is simply reported as failed and the next tick is the retry, which +/// keeps the sweep bounded in time and keeps the two-strike damping in +/// the only place that decides whether a failure is worth +/// showing. Retrying inside a sweep would silently double the effective strike count. +/// +public sealed class OverviewPollerService : BackgroundService +{ + /// Name of the pooled used for probes. + public const string HttpClientName = "overview-health"; + + /// + /// Slack allowed when deciding whether an instance is due. A timer tick can arrive a hair early + /// relative to the recorded sweep time, and without the tolerance such an instance would be + /// skipped and end up polling at half its configured rate. + /// + private static readonly TimeSpan DueTolerance = TimeSpan.FromMilliseconds(250); + + private readonly IHttpClientFactory _httpClientFactory; + private readonly OverviewSnapshotStore _store; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly OverviewMetrics _metrics; + private readonly PollCycleHeartbeat _heartbeat; + private readonly List _targets; + + /// Initializes a new . + /// The validated registry. + /// Factory for the pooled probe client. + /// The snapshot store this service writes to. + /// Clock and timer source. + /// The app's own instruments. + /// Records that a sweep completed, for the self health check. + /// Logger. + public OverviewPollerService( + IOptions options, + IHttpClientFactory httpClientFactory, + OverviewSnapshotStore store, + TimeProvider timeProvider, + OverviewMetrics metrics, + PollCycleHeartbeat heartbeat, + ILogger logger) + { + ArgumentNullException.ThrowIfNull(options); + + _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory)); + _store = store ?? throw new ArgumentNullException(nameof(store)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics)); + _heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _targets = BuildTargets(options.Value); + } + + /// + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + if (_targets.Count == 0) + { + _logger.LogWarning("Overview registry contains no instances; the poller has nothing to do."); + return; + } + + // Tick at the fastest configured cadence and let each target decide whether it is due. + // A single timer honours per-instance overrides without spawning a timer per instance. + var tick = _targets.Min(t => t.Timings.PollInterval); + + // Publish the registry before probing anything, so the page paints its full layout at once + // instead of appearing empty for the duration of the first sweep. + _store.Publish(BuildSnapshot(_timeProvider.GetUtcNow())); + + using var timer = new PeriodicTimer(tick, _timeProvider); + + try + { + do + { + await SweepAsync(stoppingToken).ConfigureAwait(false); + } + while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false)); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + // Ordinary shutdown. + } + } + + /// Runs one sweep: probes every due instance and publishes the resulting snapshot. + /// Cancelled on host shutdown. + /// The snapshot that was published. + internal async Task SweepAsync(CancellationToken cancellationToken = default) + { + // Checked here rather than left to the transport: whether an in-flight request observes the + // token is a property of the handler stack, but "a sweep that started after shutdown must + // not publish" has to hold unconditionally, or the store keeps a half-probed registry as + // its final state. + cancellationToken.ThrowIfCancellationRequested(); + + var startedAt = _timeProvider.GetTimestamp(); + var now = _timeProvider.GetUtcNow(); + var due = _targets.Where(t => IsDue(t, now)).ToList(); + + if (due.Count > 0) + { + // One client per sweep, not per probe: IHttpClientFactory pools the handler either way, + // and a single instance keeps connection reuse across the fan-out. + var client = _httpClientFactory.CreateClient(HttpClientName); + var probe = new ZbHealthReportClient(client); + + var results = await Task + .WhenAll(due.Select(target => ProbeAsync(probe, target, now, cancellationToken))) + .ConfigureAwait(false); + + for (var i = 0; i < due.Count; i++) + due[i].Last = results[i]; + } + + // Re-checked after the fan-out: a shutdown that lands mid-sweep leaves the probed targets + // in an arbitrary mix of fresh and stale results, and publishing that mix would freeze a + // misleading picture into the store for as long as the process takes to exit. + cancellationToken.ThrowIfCancellationRequested(); + + var snapshot = BuildSnapshot(now); + _store.Publish(snapshot); + + _metrics.RecordSweep(_timeProvider.GetElapsedTime(startedAt)); + + // Stamped after the publish, so the heartbeat can never claim a cycle the store did not + // receive — the health check's whole job is to notice exactly that gap. + _heartbeat.RecordCycleCompleted(_timeProvider.GetUtcNow()); + + return snapshot; + } + + private bool IsDue(PollTarget target, DateTimeOffset now) => + target.Last?.LastPolledUtc is not { } polled || now - polled + DueTolerance >= target.Timings.PollInterval; + + private static async Task ProbeAsync( + ZbHealthReportClient probe, + PollTarget target, + DateTimeOffset now, + CancellationToken cancellationToken) + { + var timeout = target.Timings.Timeout; + var ready = await probe.ProbeAsync(target.ReadyUrl, timeout, cancellationToken).ConfigureAwait(false); + + // Skip the active probe when nothing answered the ready probe: it would spend a second full + // timeout to learn what we already know, and ObserveActive maps a missing probe to Unknown + // anyway. A 503 from ready is NOT a skip — that instance is talking, and whether it is the + // active half is exactly what an operator wants to know about a sick node. + HealthProbeResult? active = null; + if (target.Instance.HasActiveRole && ready.Reachable) + active = await probe.ProbeStatusAsync(target.ActiveUrl, timeout, cancellationToken).ConfigureAwait(false); + + var observed = StatusDerivation.Observe(ready); + var (status, failures) = StatusDerivation.Damp( + target.Last?.Status, + target.Last?.ConsecutiveFailures ?? 0, + observed); + + return target.Template with + { + Status = status, + Active = StatusDerivation.ObserveActive(target.Instance.HasActiveRole, active), + ConsecutiveFailures = failures, + LastPolledUtc = now, + LastReachableUtc = ready.Reachable ? now : target.Last?.LastReachableUtc, + Latency = ready.Latency, + Error = ready.Error, + Report = ready.Report, + }; + } + + private OverviewSnapshot BuildSnapshot(DateTimeOffset now) + { + var applications = _targets + .GroupBy(t => t.Application, StringComparer.Ordinal) + .Select(group => + { + var instances = group.Select(t => t.Last ?? t.Template).ToList(); + return new ApplicationSnapshot(group.Key, instances, LeaderResolver.Resolve(instances)); + }) + .ToList(); + + return new OverviewSnapshot(now, applications); + } + + private static List BuildTargets(OverviewOptions options) + { + var targets = new List(); + + foreach (var application in options.Applications) + { + foreach (var instance in application.Instances) + { + var baseUrl = instance.BaseUrl.TrimEnd('/'); + + targets.Add(new PollTarget + { + Application = application.Name, + Instance = instance, + Timings = EffectiveTimings.Resolve(options, application, instance), + ReadyUrl = $"{baseUrl}/health/ready", + ActiveUrl = $"{baseUrl}/health/active", + Template = new InstanceSnapshot + { + Application = application.Name, + Instance = instance.Name, + Group = instance.Group, + BaseUrl = baseUrl, + ManagementUrl = instance.ManagementUrl, + ManagementLabel = application.ManagementLabel, + HasActiveRole = instance.HasActiveRole, + StaleAfter = EffectiveTimings.Resolve(options, application, instance).StaleAfter, + }, + }); + } + } + + return targets; + } + + /// One instance's fixed probe configuration plus its most recent result. + private sealed class PollTarget + { + public required string Application { get; init; } + + public required InstanceEntry Instance { get; init; } + + public required EffectiveTimings Timings { get; init; } + + public required string ReadyUrl { get; init; } + + public required string ActiveUrl { get; init; } + + /// The registry-derived fields, reused as the base of every result. + public required InstanceSnapshot Template { get; init; } + + /// + /// The last published result, or null before the first probe. Written only by the sweep + /// loop after its Task.WhenAll has completed, so it is never touched concurrently. + /// + public InstanceSnapshot? Last { get; set; } + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshot.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshot.cs new file mode 100644 index 0000000..b12d7b9 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshot.cs @@ -0,0 +1,172 @@ +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// Everything the dashboard knows about one instance, as of the last sweep that included it. +/// +/// +/// Immutable by construction: the poller builds a whole new object graph each sweep and swaps it +/// into atomically, so a render can never observe a half-updated +/// registry. +/// +public sealed record InstanceSnapshot +{ + /// Owning application's display name. + public required string Application { get; init; } + + /// This instance's display name, unique within its application. + public required string Instance { get; init; } + + /// Cluster group, or null for a standalone instance. + public string? Group { get; init; } + + /// The probed base URL, shown on the card for diagnosis. + public required string BaseUrl { get; init; } + + /// Management UI link, when the registry supplies one. + public string? ManagementUrl { get; init; } + + /// Label for , from the application entry. + public string ManagementLabel { get; init; } = "UI"; + + /// Whether this instance is half of an active/standby pair. + public bool HasActiveRole { get; init; } + + /// + /// The damped, displayable status — null until the first sweep has actually probed this + /// instance. + /// + /// + /// Null is a real, distinct state and not a stand-in for "Up": the registry is published to the + /// store before the first probe returns so the page renders its full layout immediately, and + /// showing an unmeasured instance as healthy for that half-second would be a fabricated reading. + /// + public InstanceStatus? Status { get; init; } + + /// Active/standby state from the active tier. + public ActiveState Active { get; init; } = ActiveState.NotApplicable; + + /// Consecutive failing observations, carried across sweeps to drive flap damping. + public int ConsecutiveFailures { get; init; } + + /// + /// When the sweep that produced this snapshot started; null before the first probe. Staleness + /// is measured from here, so a stuck or paused poller ages every card even though no probe + /// failed. + /// + public DateTimeOffset? LastPolledUtc { get; init; } + + /// When this instance last answered a probe at all, regardless of what it said. + public DateTimeOffset? LastReachableUtc { get; init; } + + /// Wall-clock duration of the last ready probe. + public TimeSpan Latency { get; init; } + + /// Why the last probe failed, when it did. + public string? Error { get; init; } + + /// The last parsed ready-tier body, which drives the per-check detail list. + public ZbHealthReport? Report { get; init; } + + /// This instance's effective staleness window. + public TimeSpan StaleAfter { get; init; } + + /// Whether this snapshot is too old to be trusted as current. + /// The current time. + /// True when the snapshot has aged past its staleness window. + /// + /// Evaluated at render rather than stored, because staleness is a function of the clock and not + /// of anything the poller observed — a snapshot that was fresh when written goes stale on its + /// own while nothing at all happens. + /// + public bool IsStale(DateTimeOffset now) => + Status is not null && LastPolledUtc is { } polled && now - polled > StaleAfter; +} + +/// Leader state for one cluster group, derived from what its members report. +/// The group name from the registry. +/// +/// Registry name of the instance the reported leader address resolves to, or null when no member +/// reported a leader or the address matches no registered instance. +/// +/// The reported Akka leader address, shown raw when unresolved. +/// +/// True when reachable members reported different leaders — the split-brain tell. +/// +/// Every distinct address reported, for the disagreement detail. +public sealed record GroupSnapshot( + string Name, + string? LeaderInstance, + string? LeaderAddress, + bool Disagreement, + IReadOnlyList ReportedLeaders); + +/// One application section of the dashboard. +/// The application's display name. +/// Its instances, in registry order. +/// Leader state per cluster group, in registry order. +public sealed record ApplicationSnapshot( + string Name, + IReadOnlyList Instances, + IReadOnlyList Groups) +{ + /// Finds the group state for . + /// The group name, or null for standalone instances. + /// The group snapshot, or null when there is none. + public GroupSnapshot? GroupFor(string? group) => + group is null ? null : Groups.FirstOrDefault(g => string.Equals(g.Name, group, StringComparison.Ordinal)); + + /// + /// The most severe status across this application's probed instances, for the rollup pill; + /// null while nothing has been probed yet. + /// + public InstanceStatus? Worst + { + get + { + InstanceStatus? worst = null; + foreach (var instance in Instances) + { + if (instance.Status is not { } status) + continue; + + if (worst is null || StatusSeverity.Rank(status) > StatusSeverity.Rank(worst.Value)) + worst = status; + } + + return worst; + } + } +} + +/// A complete, atomically-published view of the whole registry. +/// When the sweep that produced this snapshot started. +/// The applications, in registry order. +public sealed record OverviewSnapshot(DateTimeOffset GeneratedUtc, IReadOnlyList Applications) +{ + /// The pre-registry snapshot served before the poller has published anything. + public static OverviewSnapshot Empty { get; } = new(default, []); + + /// Every instance across every application, flattened. + public IEnumerable AllInstances => Applications.SelectMany(a => a.Instances); +} + +/// Severity ordering for rollups. +public static class StatusSeverity +{ + /// Ranks a status; higher is worse. + /// The status to rank. + /// The severity rank. + /// + /// outranks because + /// it carries strictly less information: a Down instance told us what is wrong, an Unreachable + /// one did not answer at all and could be anything up to and including gone. + /// + public static int Rank(InstanceStatus status) => status switch + { + InstanceStatus.Up => 0, + InstanceStatus.Degraded => 1, + InstanceStatus.Down => 2, + InstanceStatus.Unreachable => 3, + _ => 3, + }; +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshotStore.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshotStore.cs new file mode 100644 index 0000000..9f341f8 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/OverviewSnapshotStore.cs @@ -0,0 +1,38 @@ +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// Holds the current and notifies subscribers when it is replaced. +/// +/// +/// This is the cache-first requirement made concrete: the poller is the only writer, and every +/// render reads whatever is already here. A page load therefore paints the last known state +/// immediately and never waits on — or triggers — a probe of its own. +/// +public sealed class OverviewSnapshotStore +{ + private OverviewSnapshot _current = OverviewSnapshot.Empty; + + /// Raised after a new snapshot has been published. + /// + /// Handlers run on the poller's thread, so a Blazor subscriber must marshal onto its own + /// synchronization context (InvokeAsync(StateHasChanged)) and must unsubscribe when + /// disposed. + /// + public event Action? Changed; + + /// The current snapshot; never null. + public OverviewSnapshot Current => Volatile.Read(ref _current); + + /// Replaces the current snapshot and notifies subscribers. + /// The new snapshot. + public void Publish(OverviewSnapshot snapshot) + { + ArgumentNullException.ThrowIfNull(snapshot); + + // A single reference swap of an already-complete graph. Readers hold their own reference + // for the duration of a render, so a sweep landing mid-render cannot tear the view. + Volatile.Write(ref _current, snapshot); + + Changed?.Invoke(snapshot); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs new file mode 100644 index 0000000..470b5cf --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/PollingServiceCollectionExtensions.cs @@ -0,0 +1,40 @@ +using Microsoft.Extensions.DependencyInjection.Extensions; +using ZB.MOM.WW.Overview.Observability; + +namespace ZB.MOM.WW.Overview.Polling; + +/// Registers the polling pipeline. +public static class PollingServiceCollectionExtensions +{ + /// Adds the snapshot store, the probe client and the poller. + /// The service collection. + /// The same collection, for chaining. + public static IServiceCollection AddOverviewPolling(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.TryAddSingleton(TimeProvider.System); + services.TryAddSingleton(); + + // IMeterFactory comes from AddMetrics, which the default web host already registers; the + // call is idempotent and makes the dependency explicit for any non-web host. + services.AddMetrics(); + services.TryAddSingleton(); + services.TryAddSingleton(); + + services.AddHttpClient(OverviewPollerService.HttpClientName, client => + { + // The per-request CancellationTokenSource in ZbHealthReportClient is the only timeout + // that should ever fire. Leaving HttpClient's own 100s default in place would let it + // pre-empt a per-instance override and report "timed out after 3s" for a 100s wait. + client.Timeout = Timeout.InfiniteTimeSpan; + + // Health bodies are small and must not be served from anywhere but the instance itself. + client.DefaultRequestHeaders.CacheControl = new() { NoCache = true, NoStore = true }; + }); + + services.AddHostedService(); + + return services; + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReport.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReport.cs new file mode 100644 index 0000000..fb23d70 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReport.cs @@ -0,0 +1,84 @@ +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// The canonical ZbHealthWriter body every family app serves from MapZbHealth: +/// { status, totalDurationMs, entries: { name: { status, description, durationMs, data? } } }. +/// +/// +/// This mirrors a contract owned by ZB.MOM.WW.Health. It lives in the app for v1 by +/// deliberate decision (design §8.2): promoting it to a ZB.MOM.WW.Health.Client package is +/// worth a publish only once a second consumer exists. +/// +public sealed class ZbHealthReport +{ + /// Aggregate tier status: Healthy, Degraded or Unhealthy. + [JsonPropertyName("status")] + public string? Status { get; init; } + + /// Total time the tier's checks took, in milliseconds. + [JsonPropertyName("totalDurationMs")] + public double TotalDurationMs { get; init; } + + /// Per-check results, keyed by check name (verbatim, e.g. akka-cluster). + [JsonPropertyName("entries")] + public Dictionary Entries { get; init; } = new(StringComparer.Ordinal); +} + +/// One check's result inside a . +public sealed class ZbHealthEntry +{ + /// This check's status: Healthy, Degraded or Unhealthy. + [JsonPropertyName("status")] + public string? Status { get; init; } + + /// Human-readable detail. Always present in the canonical shape; may be JSON null. + [JsonPropertyName("description")] + public string? Description { get; init; } + + /// How long this check took, in milliseconds. + [JsonPropertyName("durationMs")] + public double DurationMs { get; init; } + + /// + /// Optional structured data the check published (ZB.MOM.WW.Health 0.2.0+). Absent entirely on + /// checks that publish none and on every pre-0.2.0 node — hence nullable, and hence the + /// dashboard must render fully (minus leader) without it. + /// + /// + /// Values stay as : this is an open contract carrying arbitrary JSON, + /// and binding it to concrete types here would break the moment a check adds a field. + /// + [JsonPropertyName("data")] + public Dictionary? Data { get; init; } + + /// + /// Reads from as a string, or null when the key is + /// absent or is not a JSON string. + /// + /// The data key, matched verbatim. + /// The string value, or null. + public string? DataString(string key) + { + if (Data is null || !Data.TryGetValue(key, out var value)) + return null; + + return value.ValueKind == JsonValueKind.String ? value.GetString() : null; + } + + /// + /// Reads from as an integer, or null when the key is + /// absent or is not a JSON number. + /// + /// The data key, matched verbatim. + /// The integer value, or null. + public int? DataInt(string key) + { + if (Data is null || !Data.TryGetValue(key, out var value)) + return null; + + return value.ValueKind == JsonValueKind.Number && value.TryGetInt32(out var parsed) ? parsed : null; + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReportClient.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReportClient.cs new file mode 100644 index 0000000..4e82cd3 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Polling/ZbHealthReportClient.cs @@ -0,0 +1,139 @@ +using System.Diagnostics; +using System.Text.Json; + +namespace ZB.MOM.WW.Overview.Polling; + +/// +/// One probe of one health endpoint: whether anything answered, what it answered, and how long it +/// took. +/// +/// +/// True when the endpoint answered with a parseable canonical body. False for timeouts, refused +/// connections, DNS failures, and bodies that are not the canonical health JSON. +/// +/// The HTTP status code, when one was received. +/// The parsed body, when it parsed. +/// Wall-clock time for the request. +/// A short reason when is false. +public sealed record HealthProbeResult( + bool Reachable, + int? StatusCode, + ZbHealthReport? Report, + TimeSpan Latency, + string? Error); + +/// +/// GETs a family health endpoint and parses the canonical ZbHealthWriter body. +/// +/// +/// Both 200 and 503 are treated as SUCCESSFUL probes: an Unhealthy tier still returns a full body, +/// and that body is exactly what the dashboard needs to show which check failed. Only a genuine +/// non-answer (transport failure) or an unparseable body is a failed probe. +/// +public sealed class ZbHealthReportClient +{ + private static readonly JsonSerializerOptions SerializerOptions = new() + { + PropertyNameCaseInsensitive = true, + }; + + private readonly HttpClient _httpClient; + + /// Initializes a new . + /// The HTTP client used for probes. + public ZbHealthReportClient(HttpClient httpClient) => + _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient)); + + /// Probes and parses the response body. + /// Absolute URL of a health endpoint. + /// Per-request timeout. + /// Token cancelled when the host is shutting down. + /// The probe result; this never throws for an unreachable endpoint. + public Task ProbeAsync( + string url, + TimeSpan timeout, + CancellationToken cancellationToken = default) => + SendAsync(url, timeout, bodyRequired: true, cancellationToken); + + /// + /// Probes for its status code alone, parsing the body only if one + /// happens to be there. + /// + /// Absolute URL of a health endpoint. + /// Per-request timeout. + /// Token cancelled when the host is shutting down. + /// The probe result; this never throws for an unreachable endpoint. + /// + /// For the active tier the status code IS the answer — 200 means active, 503 means standby — + /// and the body carries nothing the dashboard uses. Demanding a parseable body here would + /// report every node behind anything that answers the tier with an empty response (a bare + /// IActiveNodeGate endpoint, a load balancer, a proxy) as "role unknown" while it is + /// plainly telling us the role. + /// + public Task ProbeStatusAsync( + string url, + TimeSpan timeout, + CancellationToken cancellationToken = default) => + SendAsync(url, timeout, bodyRequired: false, cancellationToken); + + private async Task SendAsync( + string url, + TimeSpan timeout, + bool bodyRequired, + CancellationToken cancellationToken) + { + var startedAt = Stopwatch.GetTimestamp(); + + // Linked, not a bare timeout token: shutdown must cancel an in-flight probe immediately, + // and the two causes have to stay distinguishable so a shutdown is not misreported as a + // timed-out instance. + using var timeoutSource = new CancellationTokenSource(timeout); + using var linked = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeoutSource.Token); + + try + { + using var response = await _httpClient + .GetAsync(url, HttpCompletionOption.ResponseContentRead, linked.Token) + .ConfigureAwait(false); + + var body = await response.Content.ReadAsStringAsync(linked.Token).ConfigureAwait(false); + var statusCode = (int)response.StatusCode; + + ZbHealthReport? report; + try + { + report = JsonSerializer.Deserialize(body, SerializerOptions); + } + catch (JsonException ex) + { + // Answered, but not with this contract — a reverse proxy error page, an HTML login + // redirect, a wrong port. That is a registry/plumbing problem, not a sick app, so + // it lands as Unreachable rather than Down. + return bodyRequired + ? Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}") + : new HealthProbeResult(true, statusCode, null, Stopwatch.GetElapsedTime(startedAt), null); + } + + if (report is null && bodyRequired) + return Failed(startedAt, statusCode, "response body deserialized to null"); + + return new HealthProbeResult(true, statusCode, report, Stopwatch.GetElapsedTime(startedAt), null); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + // Host shutdown, not an instance fault — let the caller abandon the sweep. + throw; + } + catch (OperationCanceledException) + { + return Failed(startedAt, null, $"timed out after {timeout.TotalSeconds:0.##}s"); + } + catch (HttpRequestException ex) + { + return Failed(startedAt, null, ex.Message); + } + } + + private static HealthProbeResult Failed(long startedAt, int? statusCode, string error) => + new(false, statusCode, null, Stopwatch.GetElapsedTime(startedAt), error); +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs new file mode 100644 index 0000000..0d0228e --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Program.cs @@ -0,0 +1,81 @@ +using ZB.MOM.WW.Configuration; +using ZB.MOM.WW.Health; +using ZB.MOM.WW.Overview.Components; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; +using ZB.MOM.WW.Overview.Registry; +using ZB.MOM.WW.Telemetry; +using ZB.MOM.WW.Telemetry.Serilog; + +var builder = WebApplication.CreateBuilder(args); + +// The host calls this itself, but ONLY when the environment is literally "Development". Running the +// build output under any other name — checking a staging registry locally, say — leaves the Theme +// RCL's assets under _content/ unresolvable, and the page renders as a blank white screen with its +// markup fully present and every stylesheet 404. Calling it unconditionally is a no-op once +// published (the manifest is gone and the assets are real files on disk), so the only thing it +// changes is that failure. +builder.WebHost.UseStaticWebAssets(); + +// Pre-host presence check. The validator below covers shape and consistency, but it only runs once +// the options are resolved — a registry section that is missing entirely binds to an empty +// OverviewOptions, and the failure a reader can act on is "you configured no registry", not +// "Applications must have at least 1 entry". Fail here, before the host is built, with the key name. +ConfigPreflight.For(builder.Configuration) + .RequireValue($"{OverviewOptions.SectionName}:Applications:0:Name") + .RequireValue($"{OverviewOptions.SectionName}:Applications:0:Instances:0:BaseUrl") + .ThrowIfInvalid(); + +builder.Services.AddValidatedOptions( + builder.Configuration, OverviewOptions.SectionName); + +builder.AddZbSerilog(o => o.ServiceName = "overview"); + +builder.AddZbTelemetry(o => +{ + o.ServiceName = "overview"; + + // Meters is an ALLOWLIST, not a filter: a Meter absent from this array is never registered with + // the MeterProvider, and its instruments vanish from /metrics with no warning anywhere. The + // dashboard's own gauge is the only thing here that would go missing silently. + o.Meters = [OverviewMetrics.MeterName]; +}); + +builder.Services.AddOverviewPolling(); +builder.Services.AddOverviewHealthChecks(); + +builder.Services.AddRazorComponents() + .AddInteractiveServerComponents(); + +var app = builder.Build(); + +app.UseStaticFiles(); + +// No UseAuthentication/UseAuthorization and no cascading auth state: the dashboard is anonymous and +// read-only by requirement (design §2). It renders health data that every one of these apps already +// serves unauthenticated, and it has no endpoint that mutates anything. +// +// UseAntiforgery IS required despite that. AddRazorComponents stamps antiforgery metadata onto every +// component endpoint unconditionally, and the endpoint middleware hard-fails a request whose endpoint +// carries that metadata with no middleware to honour it — so without this line every page returns +// 500. The alternative, MapRazorComponents(...).DisableAntiforgery(), would also boot, but it turns +// the first form anyone ever adds into a silent CSRF hole. This app has no forms, so the middleware +// is inert; it is here so that stays true by default rather than by vigilance. +app.UseAntiforgery(); + +app.MapRazorComponents() + .AddInteractiveServerRenderMode(); + +// The dashboard's own three-tier health surface and Prometheus endpoint. Both are anonymous, like +// every /health/* endpoint this app polls — which also means one Overview instance can be pointed +// at another and would render it correctly. +app.MapZbHealth(); +app.MapZbMetrics(); + +await app.RunAsync(); + +/// +/// Exposed so WebApplicationFactory<Program> can boot the real host in tests +/// (top-level statements otherwise compile to an internal entry-point class). +/// +public partial class Program; diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptions.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptions.cs new file mode 100644 index 0000000..127aa27 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptions.cs @@ -0,0 +1,124 @@ +namespace ZB.MOM.WW.Overview.Registry; + +/// +/// The dashboard's registry: which application instances to probe, and how often. Bound from the +/// Overview configuration section and validated by . +/// +/// +/// Every instance is probed the same way — GET {BaseUrl}/health/ready, plus +/// {BaseUrl}/health/active when . There is +/// deliberately no per-instance probe-mode switch: one probe model, one status derivation. +/// +public sealed class OverviewOptions +{ + /// The configuration section this binds to. + public const string SectionName = "Overview"; + + /// How often the poller sweeps every instance, in seconds. + public int PollIntervalSeconds { get; set; } = 10; + + /// Per-request timeout, in seconds. + public int TimeoutSeconds { get; set; } = 3; + + /// + /// How old a snapshot may get before its instance renders as Stale (out of date / offline) + /// regardless of its last known status, in seconds. This is what catches a stuck poller, a + /// paused refresh or a slept machine — not just failed probes — so it must exceed the poll + /// interval or every instance would read Stale between two healthy polls. + /// + public int StaleAfterSeconds { get; set; } = 45; + + /// The registered applications, each with one or more instances. + public IList Applications { get; set; } = new List(); +} + +/// One registered application (a section on the dashboard). +public sealed class ApplicationEntry +{ + /// Display name, unique across the registry. + public string Name { get; set; } = string.Empty; + + /// Label for the per-instance management link, e.g. "AdminUI" or "Dashboard". + public string ManagementLabel { get; set; } = "UI"; + + /// Overrides for this application. + public int? PollIntervalSeconds { get; set; } + + /// Overrides for this application. + public int? TimeoutSeconds { get; set; } + + /// Overrides for this application. + public int? StaleAfterSeconds { get; set; } + + /// This application's instances; at least one is required. + public IList Instances { get; set; } = new List(); +} + +/// One probed instance (a card on the dashboard). +public sealed class InstanceEntry +{ + /// Display name, unique within its application. + public string Name { get; set; } = string.Empty; + + /// + /// The cluster group this instance belongs to (e.g. "central", "site-a"). One Akka cluster is + /// one group: leader aggregation and the split-brain check are scoped to it. Null for + /// standalone instances. + /// + public string? Group { get; set; } + + /// Absolute base URL whose /health/* endpoints are probed. + public string BaseUrl { get; set; } = string.Empty; + + /// Absolute URL of this instance's own management UI, linked from its card. + public string? ManagementUrl { get; set; } + + /// + /// Whether this instance participates in an active/standby pair. False for the single-instance + /// gateways, which have no such concept — their cards show status without an Active badge and + /// are never probed for /health/active. + /// + public bool HasActiveRole { get; set; } + + /// Overrides the application and global poll interval for this instance. + public int? PollIntervalSeconds { get; set; } + + /// Overrides the application and global request timeout for this instance. + public int? TimeoutSeconds { get; set; } + + /// Overrides the application and global staleness window for this instance. + public int? StaleAfterSeconds { get; set; } +} + +/// +/// The effective (override-resolved) timings for one instance. Instance wins over application, +/// application wins over global — resolved once here so no caller re-implements the precedence. +/// +/// Effective poll cadence. +/// Effective per-request timeout. +/// Effective staleness window. +public readonly record struct EffectiveTimings(TimeSpan PollInterval, TimeSpan Timeout, TimeSpan StaleAfter) +{ + /// Resolves the effective timings for . + /// The global registry defaults. + /// The instance's application. + /// The instance itself. + /// The resolved timings. + public static EffectiveTimings Resolve( + OverviewOptions options, + ApplicationEntry application, + InstanceEntry instance) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(application); + ArgumentNullException.ThrowIfNull(instance); + + return new EffectiveTimings( + TimeSpan.FromSeconds( + instance.PollIntervalSeconds ?? application.PollIntervalSeconds ?? options.PollIntervalSeconds), + TimeSpan.FromSeconds( + instance.TimeoutSeconds ?? application.TimeoutSeconds ?? options.TimeoutSeconds), + TimeSpan.FromSeconds( + instance.StaleAfterSeconds ?? application.StaleAfterSeconds ?? options.StaleAfterSeconds)); + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptionsValidator.cs b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptionsValidator.cs new file mode 100644 index 0000000..b6e1274 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Registry/OverviewOptionsValidator.cs @@ -0,0 +1,109 @@ +using ZB.MOM.WW.Configuration; + +namespace ZB.MOM.WW.Overview.Registry; + +/// +/// Validates the Overview registry. Registered through +/// AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>, so a malformed +/// registry fails the host at startup rather than surfacing as a silently-empty dashboard or a +/// first-poll exception. +/// +public sealed class OverviewOptionsValidator : OptionsValidatorBase +{ + /// + protected override void Validate(ValidationBuilder builder, OverviewOptions options) + { + ArgumentNullException.ThrowIfNull(builder); + ArgumentNullException.ThrowIfNull(options); + + RequirePositive(builder, options.PollIntervalSeconds, "Overview:PollIntervalSeconds"); + RequirePositive(builder, options.TimeoutSeconds, "Overview:TimeoutSeconds"); + RequirePositive(builder, options.StaleAfterSeconds, "Overview:StaleAfterSeconds"); + + builder.MinCount(options.Applications as IReadOnlyCollection, 1, "Overview:Applications"); + + var seenApplications = new HashSet(StringComparer.OrdinalIgnoreCase); + + for (var a = 0; a < options.Applications.Count; a++) + { + var application = options.Applications[a]; + var appPath = $"Overview:Applications:{a}"; + + builder.Required(application.Name, $"{appPath}:Name"); + if (!string.IsNullOrWhiteSpace(application.Name) && !seenApplications.Add(application.Name)) + builder.Add($"{appPath}:Name '{application.Name}' is duplicated; application names must be unique"); + + RequirePositiveIfSet(builder, application.PollIntervalSeconds, $"{appPath}:PollIntervalSeconds"); + RequirePositiveIfSet(builder, application.TimeoutSeconds, $"{appPath}:TimeoutSeconds"); + RequirePositiveIfSet(builder, application.StaleAfterSeconds, $"{appPath}:StaleAfterSeconds"); + + builder.MinCount( + application.Instances as IReadOnlyCollection, 1, $"{appPath}:Instances"); + + var seenInstances = new HashSet(StringComparer.OrdinalIgnoreCase); + + for (var i = 0; i < application.Instances.Count; i++) + { + var instance = application.Instances[i]; + var path = $"{appPath}:Instances:{i}"; + + builder.Required(instance.Name, $"{path}:Name"); + if (!string.IsNullOrWhiteSpace(instance.Name) && !seenInstances.Add(instance.Name)) + { + builder.Add( + $"{path}:Name '{instance.Name}' is duplicated within application " + + $"'{application.Name}'; instance names must be unique per application"); + } + + RequireAbsoluteHttpUrl(builder, instance.BaseUrl, $"{path}:BaseUrl", required: true); + RequireAbsoluteHttpUrl(builder, instance.ManagementUrl, $"{path}:ManagementUrl", required: false); + + RequirePositiveIfSet(builder, instance.PollIntervalSeconds, $"{path}:PollIntervalSeconds"); + RequirePositiveIfSet(builder, instance.TimeoutSeconds, $"{path}:TimeoutSeconds"); + RequirePositiveIfSet(builder, instance.StaleAfterSeconds, $"{path}:StaleAfterSeconds"); + + // Checked on the EFFECTIVE values, not the globals: an override at either level can + // invert the relationship on one instance while the global pair stays sane, and an + // instance whose stale window is inside its poll interval would render Stale + // between two perfectly healthy polls. + var timings = EffectiveTimings.Resolve(options, application, instance); + if (timings.StaleAfter <= timings.PollInterval) + { + builder.Add( + $"{path}: effective StaleAfterSeconds ({timings.StaleAfter.TotalSeconds:0.##}) must be " + + $"greater than effective PollIntervalSeconds ({timings.PollInterval.TotalSeconds:0.##}) " + + "— otherwise this instance renders Stale between two healthy polls"); + } + } + } + } + + private static void RequirePositive(ValidationBuilder builder, int value, string field) => + builder.RequireThat(value > 0, $"{field} must be greater than 0"); + + private static void RequirePositiveIfSet(ValidationBuilder builder, int? value, string field) + { + if (value is { } set) + RequirePositive(builder, set, field); + } + + private static void RequireAbsoluteHttpUrl( + ValidationBuilder builder, string? value, string field, bool required) + { + if (string.IsNullOrWhiteSpace(value)) + { + if (required) + builder.Add($"{field} is required"); + return; + } + + // Absolute + http(s) explicitly: a relative URL binds without complaint and only fails at + // the first poll, as an opaque HttpClient error against a dashboard that has no base + // address of its own. + if (!Uri.TryCreate(value, UriKind.Absolute, out var uri) + || (uri.Scheme != Uri.UriSchemeHttp && uri.Scheme != Uri.UriSchemeHttps)) + { + builder.Add($"{field} must be an absolute http:// or https:// URL (was '{value}')"); + } + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj new file mode 100644 index 0000000..d8df41c --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj @@ -0,0 +1,38 @@ + + + + + net10.0 + ZB.MOM.WW.Overview + zb-mom-ww-overview + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json new file mode 100644 index 0000000..3c0de0f --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Development.json @@ -0,0 +1,89 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + + // Kestrel endpoint config, NOT ASPNETCORE_URLS. Setting both puts Kestrel into + // explicit-endpoints mode and silently discards one of them — the family's recurring + // URLs-override trap. + "Kestrel": { + "Endpoints": { + "Http": { + "Url": "http://localhost:5320" + } + } + }, + + // Development registry for `dotnet run` ON THE MAC HOST. This is deliberately a SUBSET of the + // fleet: the dev rigs publish only some node ports to the host, so these are the instances a + // host-run dashboard can genuinely reach. Verified against the running containers 2026-07-24. + // + // ScadaBridge central pair localhost:9001 / :9002 (container :5000 published) + // ScadaBridge site nodes NOT published — their health port (8084) is container-internal + // OtOpcUa (all six nodes) NOT published — only Traefik's load-balanced :9200, which + // round-robins the central pair and so cannot identify a node + // HistorianGateway localhost:5220 + // MxAccessGateway windev 10.100.0.48:5130 (needs the VPN up) + // + // Run the containerised dashboard (docker/, environment "Docker") to see the whole fleet: it + // joins the rig networks and addresses every node by container DNS name. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", + "Instances": [ + { + "Name": "central-a", + "Group": "central", + "BaseUrl": "http://localhost:9001", + "ManagementUrl": "http://localhost:9001/", + "HasActiveRole": true + }, + { + "Name": "central-b", + "Group": "central", + "BaseUrl": "http://localhost:9002", + "ManagementUrl": "http://localhost:9002/", + "HasActiveRole": true + } + ] + }, + { + "Name": "HistorianGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "histgw", + "BaseUrl": "http://localhost:5220", + "ManagementUrl": "http://localhost:5220/", + "HasActiveRole": false, + + // Its readiness probe reaches the real historian over the VPN; when that is down the + // probe hangs rather than refusing, so it needs more room than the 3s default before + // it is honestly called unreachable. + "TimeoutSeconds": 10 + } + ] + }, + { + "Name": "MxAccessGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "windev", + "BaseUrl": "http://10.100.0.48:5130", + "ManagementUrl": "http://10.100.0.48:5130/", + "HasActiveRole": false + } + ] + } + ] + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json new file mode 100644 index 0000000..e8f4751 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.Docker.json @@ -0,0 +1,163 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + + "Kestrel": { + "Endpoints": { + "Http": { + "Url": "http://+:5320" + } + } + }, + + // Full-fleet registry for the CONTAINERISED dashboard, which joins the rig networks + // (otopcua-dev_default, scadabridge-net, zb-historiangw_default) and therefore reaches every + // node by container DNS name — including the ones whose ports are never published to the host. + // + // Ports below were verified by probing the running rigs on 2026-07-24, not read off the compose + // files, because the two disagree: + // OtOpcUa central-1/2 :9000 (ASPNETCORE_URLS) + // OtOpcUa site-* :8080 (NOT 9000 — the site nodes' explicit Kestrel listeners for + // LocalDb sync re-bind the primary HTTP port) + // ScadaBridge central :5000 + // ScadaBridge site :8084 (MetricsPort; MapZbHealth lands here from the site-health work) + // + // ManagementUrl values stay host-relative (localhost:*) on purpose: they are followed by the + // OPERATOR'S BROWSER, not by this container, so container DNS names would be dead links. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [ + { + "Name": "ScadaBridge", + "ManagementLabel": "CentralUI", + "Instances": [ + { + "Name": "central-a", + "Group": "central", + "BaseUrl": "http://scadabridge-central-a:5000", + "ManagementUrl": "http://localhost:9001/", + "HasActiveRole": true + }, + { + "Name": "central-b", + "Group": "central", + "BaseUrl": "http://scadabridge-central-b:5000", + "ManagementUrl": "http://localhost:9002/", + "HasActiveRole": true + }, + { + "Name": "site-a-a", + "Group": "site-a", + "BaseUrl": "http://scadabridge-site-a-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-a-b", + "Group": "site-a", + "BaseUrl": "http://scadabridge-site-a-b:8084", + "HasActiveRole": true + }, + { + "Name": "site-b-a", + "Group": "site-b", + "BaseUrl": "http://scadabridge-site-b-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-b-b", + "Group": "site-b", + "BaseUrl": "http://scadabridge-site-b-b:8084", + "HasActiveRole": true + }, + { + "Name": "site-c-a", + "Group": "site-c", + "BaseUrl": "http://scadabridge-site-c-a:8084", + "HasActiveRole": true + }, + { + "Name": "site-c-b", + "Group": "site-c", + "BaseUrl": "http://scadabridge-site-c-b:8084", + "HasActiveRole": true + } + ] + }, + { + "Name": "OtOpcUa", + "ManagementLabel": "AdminUI", + "Instances": [ + { + "Name": "central-1", + "Group": "MAIN", + "BaseUrl": "http://central-1:9000", + "ManagementUrl": "http://localhost:9200/", + "HasActiveRole": true + }, + { + "Name": "central-2", + "Group": "MAIN", + "BaseUrl": "http://central-2:9000", + "ManagementUrl": "http://localhost:9200/", + "HasActiveRole": true + }, + { + "Name": "site-a-1", + "Group": "SITE-A", + "BaseUrl": "http://site-a-1:8080", + "HasActiveRole": true + }, + { + "Name": "site-a-2", + "Group": "SITE-A", + "BaseUrl": "http://site-a-2:8080", + "HasActiveRole": true + }, + { + "Name": "site-b-1", + "Group": "SITE-B", + "BaseUrl": "http://site-b-1:8080", + "HasActiveRole": true + }, + { + "Name": "site-b-2", + "Group": "SITE-B", + "BaseUrl": "http://site-b-2:8080", + "HasActiveRole": true + } + ] + }, + { + "Name": "HistorianGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "histgw", + "BaseUrl": "http://zb-historiangw:5220", + "ManagementUrl": "http://localhost:5220/", + "HasActiveRole": false, + "TimeoutSeconds": 10 + } + ] + }, + { + "Name": "MxAccessGateway", + "ManagementLabel": "Dashboard", + "Instances": [ + { + "Name": "windev", + "BaseUrl": "http://10.100.0.48:5130", + "ManagementUrl": "http://10.100.0.48:5130/", + "HasActiveRole": false + } + ] + } + ] + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json new file mode 100644 index 0000000..f66f47d --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/appsettings.json @@ -0,0 +1,54 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "AllowedHosts": "*", + + // AddZbSerilog drives its sinks entirely from this section (ReadFrom.Configuration) and adds + // none of its own. An empty "Serilog": {} therefore produces an app that logs NOWHERE — verified + // against the running HistorianGateway container, whose log stream is completely empty for + // exactly this reason. Declaring the sinks here is what keeps this app diagnosable. + "Serilog": { + // Every probe emits four Information-level HttpClient lines. At 16 instances on a 10s cadence + // that is several lines a second of pure noise from an app whose entire job is polling — it + // would bury anything worth reading. Note this override, not Logging:LogLevel, is what governs + // once Serilog takes over. + "MinimumLevel": { + "Default": "Information", + "Override": { + "Microsoft.AspNetCore": "Warning", + "System.Net.Http.HttpClient": "Warning" + } + }, + "Using": [ + "Serilog.Sinks.Console", + "Serilog.Sinks.File" + ], + "WriteTo": [ + { "Name": "Console" }, + { + "Name": "File", + "Args": { + "path": "logs/overview-.log", + "rollingInterval": "Day" + } + } + ] + }, + + "Telemetry": {}, + + // Tuning defaults only. `Applications` is deliberately EMPTY here: the registry is a + // per-deployment fact, and shipping a placeholder fleet would let a misconfigured deployment + // boot and render instances nobody asked about. With no registry supplied, ConfigPreflight + // fails startup and names the missing key. See docker/README.md for the full shape. + "Overview": { + "PollIntervalSeconds": 10, + "TimeoutSeconds": 3, + "StaleAfterSeconds": 45, + "Applications": [] + } +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/css/site.css b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/css/site.css new file mode 100644 index 0000000..b5e29c2 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/css/site.css @@ -0,0 +1,112 @@ +/* ═══════════════════════════════════════════════════════════════════════════ + ZB.MOM.WW.Overview — app-local styles. + + Everything here is what the shared ZB.MOM.WW.Theme kit does NOT already + provide. The kit owns the design tokens, the shell, and the generic + vocabulary this file builds on: .chip-*, .panel, .panel-foot, .kv, .agg-*, + .card-grid, .conn-pill, .rail-btn, .mono, .rise. None of those are + redefined here — this file adds only the dashboard-specific anatomy from + docs/mockups/overview-dashboard-mockup.html (page-head, app/group headers, + the instance card, the check-detail list). + ═══════════════════════════════════════════════════════════════════════════ */ + +/* ── Page header ────────────────────────────────────────────────────────── */ +.page-head { display: flex; align-items: baseline; gap: 1rem; flex-wrap: wrap; margin: 0 0 1rem; } +.page-head h1 { font-size: 1.25rem; font-weight: 600; letter-spacing: 0.01em; margin: 0; text-wrap: balance; } +.page-head .spacer { flex: 1; } +.page-meta { font-family: var(--mono); font-size: 0.78rem; color: var(--ink-soft); } + +/* KPI variants the kit does not carry (it ships .alert / .caution only). */ +.agg-card.good .agg-value { color: var(--ok); } +.agg-card.quiet .agg-value { color: var(--idle); } +.kv .v.idle { color: var(--idle); } + +/* ── Application sections ───────────────────────────────────────────────── */ +.app-section { margin-bottom: 1.5rem; } +.app-head { + display: flex; align-items: baseline; gap: 0.6rem; flex-wrap: wrap; + padding-bottom: 0.4rem; border-bottom: 1px solid var(--rule-strong); margin-bottom: 0.75rem; +} +.app-head h2 { font-size: 1.02rem; font-weight: 600; margin: 0; } +.app-head .spacer { flex: 1; } +.app-meta { font-family: var(--mono); font-size: 0.76rem; color: var(--ink-faint); } + +.group { margin-bottom: 0.9rem; } +.group-head { display: flex; align-items: center; gap: 0.55rem; flex-wrap: wrap; margin-bottom: 0.5rem; } +.group-name { + font-size: 0.74rem; font-weight: 600; text-transform: uppercase; + letter-spacing: 0.07em; color: var(--ink-soft); +} +.group-meta { font-family: var(--mono); font-size: 0.74rem; color: var(--ink-faint); } + +/* ── Instance card ──────────────────────────────────────────────────────── */ +.inst { + background: var(--card); border: 1px solid var(--rule); border-radius: 8px; + overflow: hidden; display: flex; flex-direction: column; +} + +/* Status is carried by the top stripe. Unreachable additionally goes DASHED: + colour alone cannot distinguish it from Down, and the two send an operator + to completely different places (network vs. the application itself). */ +.inst.up { border-top: 3px solid var(--ok); } +.inst.degraded { border-top: 3px solid var(--warn); } +.inst.down { border-top: 3px solid var(--bad); } +.inst.unreachable { border-top: 3px solid var(--bad); border-style: dashed; border-top-style: solid; } +.inst.pending { border-top: 3px solid var(--rule-strong); } +.inst.stale { border-top: 3px solid var(--idle); } +.inst.stale .inst-body, +.inst.stale .inst-head .inst-host { opacity: 0.55; } + +.inst-head { + display: flex; align-items: flex-start; justify-content: space-between; + gap: 0.6rem; padding: 0.6rem 0.9rem 0.45rem; +} +.inst-name { font-weight: 600; font-size: 0.92rem; } +.inst-host { + display: block; font-family: var(--mono); font-size: 0.72rem; + color: var(--ink-faint); margin-top: 0.15rem; +} +.inst-chips { display: flex; gap: 0.3rem; flex-wrap: wrap; justify-content: flex-end; } + +.inst .panel-foot { + margin-top: auto; display: flex; align-items: center; + justify-content: space-between; gap: 0.6rem; +} +.foot-note { font-family: var(--mono); font-size: 0.72rem; color: var(--ink-faint); } + +/* ── Check detail (native
, no JS) ─────────────────────────────── */ +details.checks { border-top: 1px solid var(--rule); } +details.checks summary { + list-style: none; cursor: pointer; padding: 0.45rem 0.9rem; font-size: 0.72rem; + font-weight: 600; text-transform: uppercase; letter-spacing: 0.06em; + color: var(--ink-faint); user-select: none; +} +details.checks summary::-webkit-details-marker { display: none; } +details.checks summary::before { content: '\25B6'; font-size: 0.55rem; margin-right: 0.4rem; } +details.checks[open] summary::before { content: '\25BC'; } +details.checks summary:hover { color: var(--ink); } + +.check-row { display: flex; align-items: baseline; gap: 0.5rem; padding: 0.28rem 0.9rem; font-size: 0.8rem; } +.check-row:nth-child(even) { background: var(--zebra-bg); } +.check-name { font-family: var(--mono); font-size: 0.78rem; min-width: 9.5rem; } +.check-desc { color: var(--ink-soft); font-size: 0.78rem; } +.check-dot { width: 8px; height: 8px; border-radius: 2px; flex: 0 0 8px; align-self: center; } +.check-dot.ok { background: var(--ok); } +.check-dot.warn { background: var(--warn); } +.check-dot.bad { background: var(--bad); } +.check-dot.idle { background: var(--idle); } + +.check-data { + margin: 0.35rem 0.9rem 0.6rem; padding: 0.4rem 0.6rem; background: var(--zebra-bg); + border: 1px solid var(--rule); border-radius: 4px; + font-family: var(--mono); font-size: 0.72rem; color: var(--ink-soft); line-height: 1.6; + /* Scrolls inside its own box: an Akka address plus member counts is wider than a + 310px card, and letting it widen the card would break the whole grid. */ + overflow-x: auto; white-space: nowrap; +} +.check-data b { color: var(--accent-deep); font-weight: 500; } + +.rail-note { + padding: 0 0.6rem 0.6rem; font-family: var(--mono); font-size: 0.72rem; + color: var(--ink-faint); line-height: 1.5; +} diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/css/bootstrap.min.css b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/css/bootstrap.min.css new file mode 100644 index 0000000..3993414 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/css/bootstrap.min.css @@ -0,0 +1,6 @@ +@charset "UTF-8";/*! + * Bootstrap v5.3.3 (https://getbootstrap.com/) + * Copyright 2011-2024 The Bootstrap Authors + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE) + */:root,[data-bs-theme=light]{--bs-blue:#0d6efd;--bs-indigo:#6610f2;--bs-purple:#6f42c1;--bs-pink:#d63384;--bs-red:#dc3545;--bs-orange:#fd7e14;--bs-yellow:#ffc107;--bs-green:#198754;--bs-teal:#20c997;--bs-cyan:#0dcaf0;--bs-black:#000;--bs-white:#fff;--bs-gray:#6c757d;--bs-gray-dark:#343a40;--bs-gray-100:#f8f9fa;--bs-gray-200:#e9ecef;--bs-gray-300:#dee2e6;--bs-gray-400:#ced4da;--bs-gray-500:#adb5bd;--bs-gray-600:#6c757d;--bs-gray-700:#495057;--bs-gray-800:#343a40;--bs-gray-900:#212529;--bs-primary:#0d6efd;--bs-secondary:#6c757d;--bs-success:#198754;--bs-info:#0dcaf0;--bs-warning:#ffc107;--bs-danger:#dc3545;--bs-light:#f8f9fa;--bs-dark:#212529;--bs-primary-rgb:13,110,253;--bs-secondary-rgb:108,117,125;--bs-success-rgb:25,135,84;--bs-info-rgb:13,202,240;--bs-warning-rgb:255,193,7;--bs-danger-rgb:220,53,69;--bs-light-rgb:248,249,250;--bs-dark-rgb:33,37,41;--bs-primary-text-emphasis:#052c65;--bs-secondary-text-emphasis:#2b2f32;--bs-success-text-emphasis:#0a3622;--bs-info-text-emphasis:#055160;--bs-warning-text-emphasis:#664d03;--bs-danger-text-emphasis:#58151c;--bs-light-text-emphasis:#495057;--bs-dark-text-emphasis:#495057;--bs-primary-bg-subtle:#cfe2ff;--bs-secondary-bg-subtle:#e2e3e5;--bs-success-bg-subtle:#d1e7dd;--bs-info-bg-subtle:#cff4fc;--bs-warning-bg-subtle:#fff3cd;--bs-danger-bg-subtle:#f8d7da;--bs-light-bg-subtle:#fcfcfd;--bs-dark-bg-subtle:#ced4da;--bs-primary-border-subtle:#9ec5fe;--bs-secondary-border-subtle:#c4c8cb;--bs-success-border-subtle:#a3cfbb;--bs-info-border-subtle:#9eeaf9;--bs-warning-border-subtle:#ffe69c;--bs-danger-border-subtle:#f1aeb5;--bs-light-border-subtle:#e9ecef;--bs-dark-border-subtle:#adb5bd;--bs-white-rgb:255,255,255;--bs-black-rgb:0,0,0;--bs-font-sans-serif:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue","Noto Sans","Liberation Sans",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";--bs-font-monospace:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;--bs-gradient:linear-gradient(180deg, rgba(255, 255, 255, 0.15), rgba(255, 255, 255, 0));--bs-body-font-family:var(--bs-font-sans-serif);--bs-body-font-size:1rem;--bs-body-font-weight:400;--bs-body-line-height:1.5;--bs-body-color:#212529;--bs-body-color-rgb:33,37,41;--bs-body-bg:#fff;--bs-body-bg-rgb:255,255,255;--bs-emphasis-color:#000;--bs-emphasis-color-rgb:0,0,0;--bs-secondary-color:rgba(33, 37, 41, 0.75);--bs-secondary-color-rgb:33,37,41;--bs-secondary-bg:#e9ecef;--bs-secondary-bg-rgb:233,236,239;--bs-tertiary-color:rgba(33, 37, 41, 0.5);--bs-tertiary-color-rgb:33,37,41;--bs-tertiary-bg:#f8f9fa;--bs-tertiary-bg-rgb:248,249,250;--bs-heading-color:inherit;--bs-link-color:#0d6efd;--bs-link-color-rgb:13,110,253;--bs-link-decoration:underline;--bs-link-hover-color:#0a58ca;--bs-link-hover-color-rgb:10,88,202;--bs-code-color:#d63384;--bs-highlight-color:#212529;--bs-highlight-bg:#fff3cd;--bs-border-width:1px;--bs-border-style:solid;--bs-border-color:#dee2e6;--bs-border-color-translucent:rgba(0, 0, 0, 0.175);--bs-border-radius:0.375rem;--bs-border-radius-sm:0.25rem;--bs-border-radius-lg:0.5rem;--bs-border-radius-xl:1rem;--bs-border-radius-xxl:2rem;--bs-border-radius-2xl:var(--bs-border-radius-xxl);--bs-border-radius-pill:50rem;--bs-box-shadow:0 0.5rem 1rem rgba(0, 0, 0, 0.15);--bs-box-shadow-sm:0 0.125rem 0.25rem rgba(0, 0, 0, 0.075);--bs-box-shadow-lg:0 1rem 3rem rgba(0, 0, 0, 0.175);--bs-box-shadow-inset:inset 0 1px 2px rgba(0, 0, 0, 0.075);--bs-focus-ring-width:0.25rem;--bs-focus-ring-opacity:0.25;--bs-focus-ring-color:rgba(13, 110, 253, 0.25);--bs-form-valid-color:#198754;--bs-form-valid-border-color:#198754;--bs-form-invalid-color:#dc3545;--bs-form-invalid-border-color:#dc3545}[data-bs-theme=dark]{color-scheme:dark;--bs-body-color:#dee2e6;--bs-body-color-rgb:222,226,230;--bs-body-bg:#212529;--bs-body-bg-rgb:33,37,41;--bs-emphasis-color:#fff;--bs-emphasis-color-rgb:255,255,255;--bs-secondary-color:rgba(222, 226, 230, 0.75);--bs-secondary-color-rgb:222,226,230;--bs-secondary-bg:#343a40;--bs-secondary-bg-rgb:52,58,64;--bs-tertiary-color:rgba(222, 226, 230, 0.5);--bs-tertiary-color-rgb:222,226,230;--bs-tertiary-bg:#2b3035;--bs-tertiary-bg-rgb:43,48,53;--bs-primary-text-emphasis:#6ea8fe;--bs-secondary-text-emphasis:#a7acb1;--bs-success-text-emphasis:#75b798;--bs-info-text-emphasis:#6edff6;--bs-warning-text-emphasis:#ffda6a;--bs-danger-text-emphasis:#ea868f;--bs-light-text-emphasis:#f8f9fa;--bs-dark-text-emphasis:#dee2e6;--bs-primary-bg-subtle:#031633;--bs-secondary-bg-subtle:#161719;--bs-success-bg-subtle:#051b11;--bs-info-bg-subtle:#032830;--bs-warning-bg-subtle:#332701;--bs-danger-bg-subtle:#2c0b0e;--bs-light-bg-subtle:#343a40;--bs-dark-bg-subtle:#1a1d20;--bs-primary-border-subtle:#084298;--bs-secondary-border-subtle:#41464b;--bs-success-border-subtle:#0f5132;--bs-info-border-subtle:#087990;--bs-warning-border-subtle:#997404;--bs-danger-border-subtle:#842029;--bs-light-border-subtle:#495057;--bs-dark-border-subtle:#343a40;--bs-heading-color:inherit;--bs-link-color:#6ea8fe;--bs-link-hover-color:#8bb9fe;--bs-link-color-rgb:110,168,254;--bs-link-hover-color-rgb:139,185,254;--bs-code-color:#e685b5;--bs-highlight-color:#dee2e6;--bs-highlight-bg:#664d03;--bs-border-color:#495057;--bs-border-color-translucent:rgba(255, 255, 255, 0.15);--bs-form-valid-color:#75b798;--bs-form-valid-border-color:#75b798;--bs-form-invalid-color:#ea868f;--bs-form-invalid-border-color:#ea868f}*,::after,::before{box-sizing:border-box}@media (prefers-reduced-motion:no-preference){:root{scroll-behavior:smooth}}body{margin:0;font-family:var(--bs-body-font-family);font-size:var(--bs-body-font-size);font-weight:var(--bs-body-font-weight);line-height:var(--bs-body-line-height);color:var(--bs-body-color);text-align:var(--bs-body-text-align);background-color:var(--bs-body-bg);-webkit-text-size-adjust:100%;-webkit-tap-highlight-color:transparent}hr{margin:1rem 0;color:inherit;border:0;border-top:var(--bs-border-width) solid;opacity:.25}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{margin-top:0;margin-bottom:.5rem;font-weight:500;line-height:1.2;color:var(--bs-heading-color)}.h1,h1{font-size:calc(1.375rem + 1.5vw)}@media (min-width:1200px){.h1,h1{font-size:2.5rem}}.h2,h2{font-size:calc(1.325rem + .9vw)}@media (min-width:1200px){.h2,h2{font-size:2rem}}.h3,h3{font-size:calc(1.3rem + .6vw)}@media (min-width:1200px){.h3,h3{font-size:1.75rem}}.h4,h4{font-size:calc(1.275rem + .3vw)}@media (min-width:1200px){.h4,h4{font-size:1.5rem}}.h5,h5{font-size:1.25rem}.h6,h6{font-size:1rem}p{margin-top:0;margin-bottom:1rem}abbr[title]{-webkit-text-decoration:underline dotted;text-decoration:underline dotted;cursor:help;-webkit-text-decoration-skip-ink:none;text-decoration-skip-ink:none}address{margin-bottom:1rem;font-style:normal;line-height:inherit}ol,ul{padding-left:2rem}dl,ol,ul{margin-top:0;margin-bottom:1rem}ol ol,ol ul,ul ol,ul ul{margin-bottom:0}dt{font-weight:700}dd{margin-bottom:.5rem;margin-left:0}blockquote{margin:0 0 1rem}b,strong{font-weight:bolder}.small,small{font-size:.875em}.mark,mark{padding:.1875em;color:var(--bs-highlight-color);background-color:var(--bs-highlight-bg)}sub,sup{position:relative;font-size:.75em;line-height:0;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}a{color:rgba(var(--bs-link-color-rgb),var(--bs-link-opacity,1));text-decoration:underline}a:hover{--bs-link-color-rgb:var(--bs-link-hover-color-rgb)}a:not([href]):not([class]),a:not([href]):not([class]):hover{color:inherit;text-decoration:none}code,kbd,pre,samp{font-family:var(--bs-font-monospace);font-size:1em}pre{display:block;margin-top:0;margin-bottom:1rem;overflow:auto;font-size:.875em}pre code{font-size:inherit;color:inherit;word-break:normal}code{font-size:.875em;color:var(--bs-code-color);word-wrap:break-word}a>code{color:inherit}kbd{padding:.1875rem .375rem;font-size:.875em;color:var(--bs-body-bg);background-color:var(--bs-body-color);border-radius:.25rem}kbd kbd{padding:0;font-size:1em}figure{margin:0 0 1rem}img,svg{vertical-align:middle}table{caption-side:bottom;border-collapse:collapse}caption{padding-top:.5rem;padding-bottom:.5rem;color:var(--bs-secondary-color);text-align:left}th{text-align:inherit;text-align:-webkit-match-parent}tbody,td,tfoot,th,thead,tr{border-color:inherit;border-style:solid;border-width:0}label{display:inline-block}button{border-radius:0}button:focus:not(:focus-visible){outline:0}button,input,optgroup,select,textarea{margin:0;font-family:inherit;font-size:inherit;line-height:inherit}button,select{text-transform:none}[role=button]{cursor:pointer}select{word-wrap:normal}select:disabled{opacity:1}[list]:not([type=date]):not([type=datetime-local]):not([type=month]):not([type=week]):not([type=time])::-webkit-calendar-picker-indicator{display:none!important}[type=button],[type=reset],[type=submit],button{-webkit-appearance:button}[type=button]:not(:disabled),[type=reset]:not(:disabled),[type=submit]:not(:disabled),button:not(:disabled){cursor:pointer}::-moz-focus-inner{padding:0;border-style:none}textarea{resize:vertical}fieldset{min-width:0;padding:0;margin:0;border:0}legend{float:left;width:100%;padding:0;margin-bottom:.5rem;font-size:calc(1.275rem + .3vw);line-height:inherit}@media (min-width:1200px){legend{font-size:1.5rem}}legend+*{clear:left}::-webkit-datetime-edit-day-field,::-webkit-datetime-edit-fields-wrapper,::-webkit-datetime-edit-hour-field,::-webkit-datetime-edit-minute,::-webkit-datetime-edit-month-field,::-webkit-datetime-edit-text,::-webkit-datetime-edit-year-field{padding:0}::-webkit-inner-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}::-webkit-search-decoration{-webkit-appearance:none}::-webkit-color-swatch-wrapper{padding:0}::-webkit-file-upload-button{font:inherit;-webkit-appearance:button}::file-selector-button{font:inherit;-webkit-appearance:button}output{display:inline-block}iframe{border:0}summary{display:list-item;cursor:pointer}progress{vertical-align:baseline}[hidden]{display:none!important}.lead{font-size:1.25rem;font-weight:300}.display-1{font-size:calc(1.625rem + 4.5vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-1{font-size:5rem}}.display-2{font-size:calc(1.575rem + 3.9vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-2{font-size:4.5rem}}.display-3{font-size:calc(1.525rem + 3.3vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-3{font-size:4rem}}.display-4{font-size:calc(1.475rem + 2.7vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-4{font-size:3.5rem}}.display-5{font-size:calc(1.425rem + 2.1vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-5{font-size:3rem}}.display-6{font-size:calc(1.375rem + 1.5vw);font-weight:300;line-height:1.2}@media (min-width:1200px){.display-6{font-size:2.5rem}}.list-unstyled{padding-left:0;list-style:none}.list-inline{padding-left:0;list-style:none}.list-inline-item{display:inline-block}.list-inline-item:not(:last-child){margin-right:.5rem}.initialism{font-size:.875em;text-transform:uppercase}.blockquote{margin-bottom:1rem;font-size:1.25rem}.blockquote>:last-child{margin-bottom:0}.blockquote-footer{margin-top:-1rem;margin-bottom:1rem;font-size:.875em;color:#6c757d}.blockquote-footer::before{content:"— "}.img-fluid{max-width:100%;height:auto}.img-thumbnail{padding:.25rem;background-color:var(--bs-body-bg);border:var(--bs-border-width) solid var(--bs-border-color);border-radius:var(--bs-border-radius);max-width:100%;height:auto}.figure{display:inline-block}.figure-img{margin-bottom:.5rem;line-height:1}.figure-caption{font-size:.875em;color:var(--bs-secondary-color)}.container,.container-fluid,.container-lg,.container-md,.container-sm,.container-xl,.container-xxl{--bs-gutter-x:1.5rem;--bs-gutter-y:0;width:100%;padding-right:calc(var(--bs-gutter-x) * .5);padding-left:calc(var(--bs-gutter-x) * .5);margin-right:auto;margin-left:auto}@media (min-width:576px){.container,.container-sm{max-width:540px}}@media (min-width:768px){.container,.container-md,.container-sm{max-width:720px}}@media (min-width:992px){.container,.container-lg,.container-md,.container-sm{max-width:960px}}@media (min-width:1200px){.container,.container-lg,.container-md,.container-sm,.container-xl{max-width:1140px}}@media (min-width:1400px){.container,.container-lg,.container-md,.container-sm,.container-xl,.container-xxl{max-width:1320px}}:root{--bs-breakpoint-xs:0;--bs-breakpoint-sm:576px;--bs-breakpoint-md:768px;--bs-breakpoint-lg:992px;--bs-breakpoint-xl:1200px;--bs-breakpoint-xxl:1400px}.row{--bs-gutter-x:1.5rem;--bs-gutter-y:0;display:flex;flex-wrap:wrap;margin-top:calc(-1 * var(--bs-gutter-y));margin-right:calc(-.5 * var(--bs-gutter-x));margin-left:calc(-.5 * var(--bs-gutter-x))}.row>*{flex-shrink:0;width:100%;max-width:100%;padding-right:calc(var(--bs-gutter-x) * .5);padding-left:calc(var(--bs-gutter-x) * .5);margin-top:var(--bs-gutter-y)}.col{flex:1 0 0%}.row-cols-auto>*{flex:0 0 auto;width:auto}.row-cols-1>*{flex:0 0 auto;width:100%}.row-cols-2>*{flex:0 0 auto;width:50%}.row-cols-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-4>*{flex:0 0 auto;width:25%}.row-cols-5>*{flex:0 0 auto;width:20%}.row-cols-6>*{flex:0 0 auto;width:16.66666667%}.col-auto{flex:0 0 auto;width:auto}.col-1{flex:0 0 auto;width:8.33333333%}.col-2{flex:0 0 auto;width:16.66666667%}.col-3{flex:0 0 auto;width:25%}.col-4{flex:0 0 auto;width:33.33333333%}.col-5{flex:0 0 auto;width:41.66666667%}.col-6{flex:0 0 auto;width:50%}.col-7{flex:0 0 auto;width:58.33333333%}.col-8{flex:0 0 auto;width:66.66666667%}.col-9{flex:0 0 auto;width:75%}.col-10{flex:0 0 auto;width:83.33333333%}.col-11{flex:0 0 auto;width:91.66666667%}.col-12{flex:0 0 auto;width:100%}.offset-1{margin-left:8.33333333%}.offset-2{margin-left:16.66666667%}.offset-3{margin-left:25%}.offset-4{margin-left:33.33333333%}.offset-5{margin-left:41.66666667%}.offset-6{margin-left:50%}.offset-7{margin-left:58.33333333%}.offset-8{margin-left:66.66666667%}.offset-9{margin-left:75%}.offset-10{margin-left:83.33333333%}.offset-11{margin-left:91.66666667%}.g-0,.gx-0{--bs-gutter-x:0}.g-0,.gy-0{--bs-gutter-y:0}.g-1,.gx-1{--bs-gutter-x:0.25rem}.g-1,.gy-1{--bs-gutter-y:0.25rem}.g-2,.gx-2{--bs-gutter-x:0.5rem}.g-2,.gy-2{--bs-gutter-y:0.5rem}.g-3,.gx-3{--bs-gutter-x:1rem}.g-3,.gy-3{--bs-gutter-y:1rem}.g-4,.gx-4{--bs-gutter-x:1.5rem}.g-4,.gy-4{--bs-gutter-y:1.5rem}.g-5,.gx-5{--bs-gutter-x:3rem}.g-5,.gy-5{--bs-gutter-y:3rem}@media (min-width:576px){.col-sm{flex:1 0 0%}.row-cols-sm-auto>*{flex:0 0 auto;width:auto}.row-cols-sm-1>*{flex:0 0 auto;width:100%}.row-cols-sm-2>*{flex:0 0 auto;width:50%}.row-cols-sm-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-sm-4>*{flex:0 0 auto;width:25%}.row-cols-sm-5>*{flex:0 0 auto;width:20%}.row-cols-sm-6>*{flex:0 0 auto;width:16.66666667%}.col-sm-auto{flex:0 0 auto;width:auto}.col-sm-1{flex:0 0 auto;width:8.33333333%}.col-sm-2{flex:0 0 auto;width:16.66666667%}.col-sm-3{flex:0 0 auto;width:25%}.col-sm-4{flex:0 0 auto;width:33.33333333%}.col-sm-5{flex:0 0 auto;width:41.66666667%}.col-sm-6{flex:0 0 auto;width:50%}.col-sm-7{flex:0 0 auto;width:58.33333333%}.col-sm-8{flex:0 0 auto;width:66.66666667%}.col-sm-9{flex:0 0 auto;width:75%}.col-sm-10{flex:0 0 auto;width:83.33333333%}.col-sm-11{flex:0 0 auto;width:91.66666667%}.col-sm-12{flex:0 0 auto;width:100%}.offset-sm-0{margin-left:0}.offset-sm-1{margin-left:8.33333333%}.offset-sm-2{margin-left:16.66666667%}.offset-sm-3{margin-left:25%}.offset-sm-4{margin-left:33.33333333%}.offset-sm-5{margin-left:41.66666667%}.offset-sm-6{margin-left:50%}.offset-sm-7{margin-left:58.33333333%}.offset-sm-8{margin-left:66.66666667%}.offset-sm-9{margin-left:75%}.offset-sm-10{margin-left:83.33333333%}.offset-sm-11{margin-left:91.66666667%}.g-sm-0,.gx-sm-0{--bs-gutter-x:0}.g-sm-0,.gy-sm-0{--bs-gutter-y:0}.g-sm-1,.gx-sm-1{--bs-gutter-x:0.25rem}.g-sm-1,.gy-sm-1{--bs-gutter-y:0.25rem}.g-sm-2,.gx-sm-2{--bs-gutter-x:0.5rem}.g-sm-2,.gy-sm-2{--bs-gutter-y:0.5rem}.g-sm-3,.gx-sm-3{--bs-gutter-x:1rem}.g-sm-3,.gy-sm-3{--bs-gutter-y:1rem}.g-sm-4,.gx-sm-4{--bs-gutter-x:1.5rem}.g-sm-4,.gy-sm-4{--bs-gutter-y:1.5rem}.g-sm-5,.gx-sm-5{--bs-gutter-x:3rem}.g-sm-5,.gy-sm-5{--bs-gutter-y:3rem}}@media (min-width:768px){.col-md{flex:1 0 0%}.row-cols-md-auto>*{flex:0 0 auto;width:auto}.row-cols-md-1>*{flex:0 0 auto;width:100%}.row-cols-md-2>*{flex:0 0 auto;width:50%}.row-cols-md-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-md-4>*{flex:0 0 auto;width:25%}.row-cols-md-5>*{flex:0 0 auto;width:20%}.row-cols-md-6>*{flex:0 0 auto;width:16.66666667%}.col-md-auto{flex:0 0 auto;width:auto}.col-md-1{flex:0 0 auto;width:8.33333333%}.col-md-2{flex:0 0 auto;width:16.66666667%}.col-md-3{flex:0 0 auto;width:25%}.col-md-4{flex:0 0 auto;width:33.33333333%}.col-md-5{flex:0 0 auto;width:41.66666667%}.col-md-6{flex:0 0 auto;width:50%}.col-md-7{flex:0 0 auto;width:58.33333333%}.col-md-8{flex:0 0 auto;width:66.66666667%}.col-md-9{flex:0 0 auto;width:75%}.col-md-10{flex:0 0 auto;width:83.33333333%}.col-md-11{flex:0 0 auto;width:91.66666667%}.col-md-12{flex:0 0 auto;width:100%}.offset-md-0{margin-left:0}.offset-md-1{margin-left:8.33333333%}.offset-md-2{margin-left:16.66666667%}.offset-md-3{margin-left:25%}.offset-md-4{margin-left:33.33333333%}.offset-md-5{margin-left:41.66666667%}.offset-md-6{margin-left:50%}.offset-md-7{margin-left:58.33333333%}.offset-md-8{margin-left:66.66666667%}.offset-md-9{margin-left:75%}.offset-md-10{margin-left:83.33333333%}.offset-md-11{margin-left:91.66666667%}.g-md-0,.gx-md-0{--bs-gutter-x:0}.g-md-0,.gy-md-0{--bs-gutter-y:0}.g-md-1,.gx-md-1{--bs-gutter-x:0.25rem}.g-md-1,.gy-md-1{--bs-gutter-y:0.25rem}.g-md-2,.gx-md-2{--bs-gutter-x:0.5rem}.g-md-2,.gy-md-2{--bs-gutter-y:0.5rem}.g-md-3,.gx-md-3{--bs-gutter-x:1rem}.g-md-3,.gy-md-3{--bs-gutter-y:1rem}.g-md-4,.gx-md-4{--bs-gutter-x:1.5rem}.g-md-4,.gy-md-4{--bs-gutter-y:1.5rem}.g-md-5,.gx-md-5{--bs-gutter-x:3rem}.g-md-5,.gy-md-5{--bs-gutter-y:3rem}}@media (min-width:992px){.col-lg{flex:1 0 0%}.row-cols-lg-auto>*{flex:0 0 auto;width:auto}.row-cols-lg-1>*{flex:0 0 auto;width:100%}.row-cols-lg-2>*{flex:0 0 auto;width:50%}.row-cols-lg-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-lg-4>*{flex:0 0 auto;width:25%}.row-cols-lg-5>*{flex:0 0 auto;width:20%}.row-cols-lg-6>*{flex:0 0 auto;width:16.66666667%}.col-lg-auto{flex:0 0 auto;width:auto}.col-lg-1{flex:0 0 auto;width:8.33333333%}.col-lg-2{flex:0 0 auto;width:16.66666667%}.col-lg-3{flex:0 0 auto;width:25%}.col-lg-4{flex:0 0 auto;width:33.33333333%}.col-lg-5{flex:0 0 auto;width:41.66666667%}.col-lg-6{flex:0 0 auto;width:50%}.col-lg-7{flex:0 0 auto;width:58.33333333%}.col-lg-8{flex:0 0 auto;width:66.66666667%}.col-lg-9{flex:0 0 auto;width:75%}.col-lg-10{flex:0 0 auto;width:83.33333333%}.col-lg-11{flex:0 0 auto;width:91.66666667%}.col-lg-12{flex:0 0 auto;width:100%}.offset-lg-0{margin-left:0}.offset-lg-1{margin-left:8.33333333%}.offset-lg-2{margin-left:16.66666667%}.offset-lg-3{margin-left:25%}.offset-lg-4{margin-left:33.33333333%}.offset-lg-5{margin-left:41.66666667%}.offset-lg-6{margin-left:50%}.offset-lg-7{margin-left:58.33333333%}.offset-lg-8{margin-left:66.66666667%}.offset-lg-9{margin-left:75%}.offset-lg-10{margin-left:83.33333333%}.offset-lg-11{margin-left:91.66666667%}.g-lg-0,.gx-lg-0{--bs-gutter-x:0}.g-lg-0,.gy-lg-0{--bs-gutter-y:0}.g-lg-1,.gx-lg-1{--bs-gutter-x:0.25rem}.g-lg-1,.gy-lg-1{--bs-gutter-y:0.25rem}.g-lg-2,.gx-lg-2{--bs-gutter-x:0.5rem}.g-lg-2,.gy-lg-2{--bs-gutter-y:0.5rem}.g-lg-3,.gx-lg-3{--bs-gutter-x:1rem}.g-lg-3,.gy-lg-3{--bs-gutter-y:1rem}.g-lg-4,.gx-lg-4{--bs-gutter-x:1.5rem}.g-lg-4,.gy-lg-4{--bs-gutter-y:1.5rem}.g-lg-5,.gx-lg-5{--bs-gutter-x:3rem}.g-lg-5,.gy-lg-5{--bs-gutter-y:3rem}}@media (min-width:1200px){.col-xl{flex:1 0 0%}.row-cols-xl-auto>*{flex:0 0 auto;width:auto}.row-cols-xl-1>*{flex:0 0 auto;width:100%}.row-cols-xl-2>*{flex:0 0 auto;width:50%}.row-cols-xl-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-xl-4>*{flex:0 0 auto;width:25%}.row-cols-xl-5>*{flex:0 0 auto;width:20%}.row-cols-xl-6>*{flex:0 0 auto;width:16.66666667%}.col-xl-auto{flex:0 0 auto;width:auto}.col-xl-1{flex:0 0 auto;width:8.33333333%}.col-xl-2{flex:0 0 auto;width:16.66666667%}.col-xl-3{flex:0 0 auto;width:25%}.col-xl-4{flex:0 0 auto;width:33.33333333%}.col-xl-5{flex:0 0 auto;width:41.66666667%}.col-xl-6{flex:0 0 auto;width:50%}.col-xl-7{flex:0 0 auto;width:58.33333333%}.col-xl-8{flex:0 0 auto;width:66.66666667%}.col-xl-9{flex:0 0 auto;width:75%}.col-xl-10{flex:0 0 auto;width:83.33333333%}.col-xl-11{flex:0 0 auto;width:91.66666667%}.col-xl-12{flex:0 0 auto;width:100%}.offset-xl-0{margin-left:0}.offset-xl-1{margin-left:8.33333333%}.offset-xl-2{margin-left:16.66666667%}.offset-xl-3{margin-left:25%}.offset-xl-4{margin-left:33.33333333%}.offset-xl-5{margin-left:41.66666667%}.offset-xl-6{margin-left:50%}.offset-xl-7{margin-left:58.33333333%}.offset-xl-8{margin-left:66.66666667%}.offset-xl-9{margin-left:75%}.offset-xl-10{margin-left:83.33333333%}.offset-xl-11{margin-left:91.66666667%}.g-xl-0,.gx-xl-0{--bs-gutter-x:0}.g-xl-0,.gy-xl-0{--bs-gutter-y:0}.g-xl-1,.gx-xl-1{--bs-gutter-x:0.25rem}.g-xl-1,.gy-xl-1{--bs-gutter-y:0.25rem}.g-xl-2,.gx-xl-2{--bs-gutter-x:0.5rem}.g-xl-2,.gy-xl-2{--bs-gutter-y:0.5rem}.g-xl-3,.gx-xl-3{--bs-gutter-x:1rem}.g-xl-3,.gy-xl-3{--bs-gutter-y:1rem}.g-xl-4,.gx-xl-4{--bs-gutter-x:1.5rem}.g-xl-4,.gy-xl-4{--bs-gutter-y:1.5rem}.g-xl-5,.gx-xl-5{--bs-gutter-x:3rem}.g-xl-5,.gy-xl-5{--bs-gutter-y:3rem}}@media (min-width:1400px){.col-xxl{flex:1 0 0%}.row-cols-xxl-auto>*{flex:0 0 auto;width:auto}.row-cols-xxl-1>*{flex:0 0 auto;width:100%}.row-cols-xxl-2>*{flex:0 0 auto;width:50%}.row-cols-xxl-3>*{flex:0 0 auto;width:33.33333333%}.row-cols-xxl-4>*{flex:0 0 auto;width:25%}.row-cols-xxl-5>*{flex:0 0 auto;width:20%}.row-cols-xxl-6>*{flex:0 0 auto;width:16.66666667%}.col-xxl-auto{flex:0 0 auto;width:auto}.col-xxl-1{flex:0 0 auto;width:8.33333333%}.col-xxl-2{flex:0 0 auto;width:16.66666667%}.col-xxl-3{flex:0 0 auto;width:25%}.col-xxl-4{flex:0 0 auto;width:33.33333333%}.col-xxl-5{flex:0 0 auto;width:41.66666667%}.col-xxl-6{flex:0 0 auto;width:50%}.col-xxl-7{flex:0 0 auto;width:58.33333333%}.col-xxl-8{flex:0 0 auto;width:66.66666667%}.col-xxl-9{flex:0 0 auto;width:75%}.col-xxl-10{flex:0 0 auto;width:83.33333333%}.col-xxl-11{flex:0 0 auto;width:91.66666667%}.col-xxl-12{flex:0 0 auto;width:100%}.offset-xxl-0{margin-left:0}.offset-xxl-1{margin-left:8.33333333%}.offset-xxl-2{margin-left:16.66666667%}.offset-xxl-3{margin-left:25%}.offset-xxl-4{margin-left:33.33333333%}.offset-xxl-5{margin-left:41.66666667%}.offset-xxl-6{margin-left:50%}.offset-xxl-7{margin-left:58.33333333%}.offset-xxl-8{margin-left:66.66666667%}.offset-xxl-9{margin-left:75%}.offset-xxl-10{margin-left:83.33333333%}.offset-xxl-11{margin-left:91.66666667%}.g-xxl-0,.gx-xxl-0{--bs-gutter-x:0}.g-xxl-0,.gy-xxl-0{--bs-gutter-y:0}.g-xxl-1,.gx-xxl-1{--bs-gutter-x:0.25rem}.g-xxl-1,.gy-xxl-1{--bs-gutter-y:0.25rem}.g-xxl-2,.gx-xxl-2{--bs-gutter-x:0.5rem}.g-xxl-2,.gy-xxl-2{--bs-gutter-y:0.5rem}.g-xxl-3,.gx-xxl-3{--bs-gutter-x:1rem}.g-xxl-3,.gy-xxl-3{--bs-gutter-y:1rem}.g-xxl-4,.gx-xxl-4{--bs-gutter-x:1.5rem}.g-xxl-4,.gy-xxl-4{--bs-gutter-y:1.5rem}.g-xxl-5,.gx-xxl-5{--bs-gutter-x:3rem}.g-xxl-5,.gy-xxl-5{--bs-gutter-y:3rem}}.table{--bs-table-color-type:initial;--bs-table-bg-type:initial;--bs-table-color-state:initial;--bs-table-bg-state:initial;--bs-table-color:var(--bs-emphasis-color);--bs-table-bg:var(--bs-body-bg);--bs-table-border-color:var(--bs-border-color);--bs-table-accent-bg:transparent;--bs-table-striped-color:var(--bs-emphasis-color);--bs-table-striped-bg:rgba(var(--bs-emphasis-color-rgb), 0.05);--bs-table-active-color:var(--bs-emphasis-color);--bs-table-active-bg:rgba(var(--bs-emphasis-color-rgb), 0.1);--bs-table-hover-color:var(--bs-emphasis-color);--bs-table-hover-bg:rgba(var(--bs-emphasis-color-rgb), 0.075);width:100%;margin-bottom:1rem;vertical-align:top;border-color:var(--bs-table-border-color)}.table>:not(caption)>*>*{padding:.5rem .5rem;color:var(--bs-table-color-state,var(--bs-table-color-type,var(--bs-table-color)));background-color:var(--bs-table-bg);border-bottom-width:var(--bs-border-width);box-shadow:inset 0 0 0 9999px var(--bs-table-bg-state,var(--bs-table-bg-type,var(--bs-table-accent-bg)))}.table>tbody{vertical-align:inherit}.table>thead{vertical-align:bottom}.table-group-divider{border-top:calc(var(--bs-border-width) * 2) solid currentcolor}.caption-top{caption-side:top}.table-sm>:not(caption)>*>*{padding:.25rem .25rem}.table-bordered>:not(caption)>*{border-width:var(--bs-border-width) 0}.table-bordered>:not(caption)>*>*{border-width:0 var(--bs-border-width)}.table-borderless>:not(caption)>*>*{border-bottom-width:0}.table-borderless>:not(:first-child){border-top-width:0}.table-striped>tbody>tr:nth-of-type(odd)>*{--bs-table-color-type:var(--bs-table-striped-color);--bs-table-bg-type:var(--bs-table-striped-bg)}.table-striped-columns>:not(caption)>tr>:nth-child(2n){--bs-table-color-type:var(--bs-table-striped-color);--bs-table-bg-type:var(--bs-table-striped-bg)}.table-active{--bs-table-color-state:var(--bs-table-active-color);--bs-table-bg-state:var(--bs-table-active-bg)}.table-hover>tbody>tr:hover>*{--bs-table-color-state:var(--bs-table-hover-color);--bs-table-bg-state:var(--bs-table-hover-bg)}.table-primary{--bs-table-color:#000;--bs-table-bg:#cfe2ff;--bs-table-border-color:#a6b5cc;--bs-table-striped-bg:#c5d7f2;--bs-table-striped-color:#000;--bs-table-active-bg:#bacbe6;--bs-table-active-color:#000;--bs-table-hover-bg:#bfd1ec;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-secondary{--bs-table-color:#000;--bs-table-bg:#e2e3e5;--bs-table-border-color:#b5b6b7;--bs-table-striped-bg:#d7d8da;--bs-table-striped-color:#000;--bs-table-active-bg:#cbccce;--bs-table-active-color:#000;--bs-table-hover-bg:#d1d2d4;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-success{--bs-table-color:#000;--bs-table-bg:#d1e7dd;--bs-table-border-color:#a7b9b1;--bs-table-striped-bg:#c7dbd2;--bs-table-striped-color:#000;--bs-table-active-bg:#bcd0c7;--bs-table-active-color:#000;--bs-table-hover-bg:#c1d6cc;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-info{--bs-table-color:#000;--bs-table-bg:#cff4fc;--bs-table-border-color:#a6c3ca;--bs-table-striped-bg:#c5e8ef;--bs-table-striped-color:#000;--bs-table-active-bg:#badce3;--bs-table-active-color:#000;--bs-table-hover-bg:#bfe2e9;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-warning{--bs-table-color:#000;--bs-table-bg:#fff3cd;--bs-table-border-color:#ccc2a4;--bs-table-striped-bg:#f2e7c3;--bs-table-striped-color:#000;--bs-table-active-bg:#e6dbb9;--bs-table-active-color:#000;--bs-table-hover-bg:#ece1be;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-danger{--bs-table-color:#000;--bs-table-bg:#f8d7da;--bs-table-border-color:#c6acae;--bs-table-striped-bg:#eccccf;--bs-table-striped-color:#000;--bs-table-active-bg:#dfc2c4;--bs-table-active-color:#000;--bs-table-hover-bg:#e5c7ca;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-light{--bs-table-color:#000;--bs-table-bg:#f8f9fa;--bs-table-border-color:#c6c7c8;--bs-table-striped-bg:#ecedee;--bs-table-striped-color:#000;--bs-table-active-bg:#dfe0e1;--bs-table-active-color:#000;--bs-table-hover-bg:#e5e6e7;--bs-table-hover-color:#000;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-dark{--bs-table-color:#fff;--bs-table-bg:#212529;--bs-table-border-color:#4d5154;--bs-table-striped-bg:#2c3034;--bs-table-striped-color:#fff;--bs-table-active-bg:#373b3e;--bs-table-active-color:#fff;--bs-table-hover-bg:#323539;--bs-table-hover-color:#fff;color:var(--bs-table-color);border-color:var(--bs-table-border-color)}.table-responsive{overflow-x:auto;-webkit-overflow-scrolling:touch}@media (max-width:575.98px){.table-responsive-sm{overflow-x:auto;-webkit-overflow-scrolling:touch}}@media (max-width:767.98px){.table-responsive-md{overflow-x:auto;-webkit-overflow-scrolling:touch}}@media (max-width:991.98px){.table-responsive-lg{overflow-x:auto;-webkit-overflow-scrolling:touch}}@media (max-width:1199.98px){.table-responsive-xl{overflow-x:auto;-webkit-overflow-scrolling:touch}}@media (max-width:1399.98px){.table-responsive-xxl{overflow-x:auto;-webkit-overflow-scrolling:touch}}.form-label{margin-bottom:.5rem}.col-form-label{padding-top:calc(.375rem + var(--bs-border-width));padding-bottom:calc(.375rem + var(--bs-border-width));margin-bottom:0;font-size:inherit;line-height:1.5}.col-form-label-lg{padding-top:calc(.5rem + var(--bs-border-width));padding-bottom:calc(.5rem + var(--bs-border-width));font-size:1.25rem}.col-form-label-sm{padding-top:calc(.25rem + var(--bs-border-width));padding-bottom:calc(.25rem + var(--bs-border-width));font-size:.875rem}.form-text{margin-top:.25rem;font-size:.875em;color:var(--bs-secondary-color)}.form-control{display:block;width:100%;padding:.375rem .75rem;font-size:1rem;font-weight:400;line-height:1.5;color:var(--bs-body-color);-webkit-appearance:none;-moz-appearance:none;appearance:none;background-color:var(--bs-body-bg);background-clip:padding-box;border:var(--bs-border-width) solid var(--bs-border-color);border-radius:var(--bs-border-radius);transition:border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-control{transition:none}}.form-control[type=file]{overflow:hidden}.form-control[type=file]:not(:disabled):not([readonly]){cursor:pointer}.form-control:focus{color:var(--bs-body-color);background-color:var(--bs-body-bg);border-color:#86b7fe;outline:0;box-shadow:0 0 0 .25rem rgba(13,110,253,.25)}.form-control::-webkit-date-and-time-value{min-width:85px;height:1.5em;margin:0}.form-control::-webkit-datetime-edit{display:block;padding:0}.form-control::-moz-placeholder{color:var(--bs-secondary-color);opacity:1}.form-control::placeholder{color:var(--bs-secondary-color);opacity:1}.form-control:disabled{background-color:var(--bs-secondary-bg);opacity:1}.form-control::-webkit-file-upload-button{padding:.375rem .75rem;margin:-.375rem -.75rem;-webkit-margin-end:.75rem;margin-inline-end:.75rem;color:var(--bs-body-color);background-color:var(--bs-tertiary-bg);pointer-events:none;border-color:inherit;border-style:solid;border-width:0;border-inline-end-width:var(--bs-border-width);border-radius:0;-webkit-transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}.form-control::file-selector-button{padding:.375rem .75rem;margin:-.375rem -.75rem;-webkit-margin-end:.75rem;margin-inline-end:.75rem;color:var(--bs-body-color);background-color:var(--bs-tertiary-bg);pointer-events:none;border-color:inherit;border-style:solid;border-width:0;border-inline-end-width:var(--bs-border-width);border-radius:0;transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-control::-webkit-file-upload-button{-webkit-transition:none;transition:none}.form-control::file-selector-button{transition:none}}.form-control:hover:not(:disabled):not([readonly])::-webkit-file-upload-button{background-color:var(--bs-secondary-bg)}.form-control:hover:not(:disabled):not([readonly])::file-selector-button{background-color:var(--bs-secondary-bg)}.form-control-plaintext{display:block;width:100%;padding:.375rem 0;margin-bottom:0;line-height:1.5;color:var(--bs-body-color);background-color:transparent;border:solid transparent;border-width:var(--bs-border-width) 0}.form-control-plaintext:focus{outline:0}.form-control-plaintext.form-control-lg,.form-control-plaintext.form-control-sm{padding-right:0;padding-left:0}.form-control-sm{min-height:calc(1.5em + .5rem + calc(var(--bs-border-width) * 2));padding:.25rem .5rem;font-size:.875rem;border-radius:var(--bs-border-radius-sm)}.form-control-sm::-webkit-file-upload-button{padding:.25rem .5rem;margin:-.25rem -.5rem;-webkit-margin-end:.5rem;margin-inline-end:.5rem}.form-control-sm::file-selector-button{padding:.25rem .5rem;margin:-.25rem -.5rem;-webkit-margin-end:.5rem;margin-inline-end:.5rem}.form-control-lg{min-height:calc(1.5em + 1rem + calc(var(--bs-border-width) * 2));padding:.5rem 1rem;font-size:1.25rem;border-radius:var(--bs-border-radius-lg)}.form-control-lg::-webkit-file-upload-button{padding:.5rem 1rem;margin:-.5rem -1rem;-webkit-margin-end:1rem;margin-inline-end:1rem}.form-control-lg::file-selector-button{padding:.5rem 1rem;margin:-.5rem -1rem;-webkit-margin-end:1rem;margin-inline-end:1rem}textarea.form-control{min-height:calc(1.5em + .75rem + calc(var(--bs-border-width) * 2))}textarea.form-control-sm{min-height:calc(1.5em + .5rem + calc(var(--bs-border-width) * 2))}textarea.form-control-lg{min-height:calc(1.5em + 1rem + calc(var(--bs-border-width) * 2))}.form-control-color{width:3rem;height:calc(1.5em + .75rem + calc(var(--bs-border-width) * 2));padding:.375rem}.form-control-color:not(:disabled):not([readonly]){cursor:pointer}.form-control-color::-moz-color-swatch{border:0!important;border-radius:var(--bs-border-radius)}.form-control-color::-webkit-color-swatch{border:0!important;border-radius:var(--bs-border-radius)}.form-control-color.form-control-sm{height:calc(1.5em + .5rem + calc(var(--bs-border-width) * 2))}.form-control-color.form-control-lg{height:calc(1.5em + 1rem + calc(var(--bs-border-width) * 2))}.form-select{--bs-form-select-bg-img:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3e%3cpath fill='none' stroke='%23343a40' stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='m2 5 6 6 6-6'/%3e%3c/svg%3e");display:block;width:100%;padding:.375rem 2.25rem .375rem .75rem;font-size:1rem;font-weight:400;line-height:1.5;color:var(--bs-body-color);-webkit-appearance:none;-moz-appearance:none;appearance:none;background-color:var(--bs-body-bg);background-image:var(--bs-form-select-bg-img),var(--bs-form-select-bg-icon,none);background-repeat:no-repeat;background-position:right .75rem center;background-size:16px 12px;border:var(--bs-border-width) solid var(--bs-border-color);border-radius:var(--bs-border-radius);transition:border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-select{transition:none}}.form-select:focus{border-color:#86b7fe;outline:0;box-shadow:0 0 0 .25rem rgba(13,110,253,.25)}.form-select[multiple],.form-select[size]:not([size="1"]){padding-right:.75rem;background-image:none}.form-select:disabled{background-color:var(--bs-secondary-bg)}.form-select:-moz-focusring{color:transparent;text-shadow:0 0 0 var(--bs-body-color)}.form-select-sm{padding-top:.25rem;padding-bottom:.25rem;padding-left:.5rem;font-size:.875rem;border-radius:var(--bs-border-radius-sm)}.form-select-lg{padding-top:.5rem;padding-bottom:.5rem;padding-left:1rem;font-size:1.25rem;border-radius:var(--bs-border-radius-lg)}[data-bs-theme=dark] .form-select{--bs-form-select-bg-img:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3e%3cpath fill='none' stroke='%23dee2e6' stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='m2 5 6 6 6-6'/%3e%3c/svg%3e")}.form-check{display:block;min-height:1.5rem;padding-left:1.5em;margin-bottom:.125rem}.form-check .form-check-input{float:left;margin-left:-1.5em}.form-check-reverse{padding-right:1.5em;padding-left:0;text-align:right}.form-check-reverse .form-check-input{float:right;margin-right:-1.5em;margin-left:0}.form-check-input{--bs-form-check-bg:var(--bs-body-bg);flex-shrink:0;width:1em;height:1em;margin-top:.25em;vertical-align:top;-webkit-appearance:none;-moz-appearance:none;appearance:none;background-color:var(--bs-form-check-bg);background-image:var(--bs-form-check-bg-image);background-repeat:no-repeat;background-position:center;background-size:contain;border:var(--bs-border-width) solid var(--bs-border-color);-webkit-print-color-adjust:exact;color-adjust:exact;print-color-adjust:exact}.form-check-input[type=checkbox]{border-radius:.25em}.form-check-input[type=radio]{border-radius:50%}.form-check-input:active{filter:brightness(90%)}.form-check-input:focus{border-color:#86b7fe;outline:0;box-shadow:0 0 0 .25rem rgba(13,110,253,.25)}.form-check-input:checked{background-color:#0d6efd;border-color:#0d6efd}.form-check-input:checked[type=checkbox]{--bs-form-check-bg-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20'%3e%3cpath fill='none' stroke='%23fff' stroke-linecap='round' stroke-linejoin='round' stroke-width='3' d='m6 10 3 3 6-6'/%3e%3c/svg%3e")}.form-check-input:checked[type=radio]{--bs-form-check-bg-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3e%3ccircle r='2' fill='%23fff'/%3e%3c/svg%3e")}.form-check-input[type=checkbox]:indeterminate{background-color:#0d6efd;border-color:#0d6efd;--bs-form-check-bg-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20'%3e%3cpath fill='none' stroke='%23fff' stroke-linecap='round' stroke-linejoin='round' stroke-width='3' d='M6 10h8'/%3e%3c/svg%3e")}.form-check-input:disabled{pointer-events:none;filter:none;opacity:.5}.form-check-input:disabled~.form-check-label,.form-check-input[disabled]~.form-check-label{cursor:default;opacity:.5}.form-switch{padding-left:2.5em}.form-switch .form-check-input{--bs-form-switch-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3e%3ccircle r='3' fill='rgba%280, 0, 0, 0.25%29'/%3e%3c/svg%3e");width:2em;margin-left:-2.5em;background-image:var(--bs-form-switch-bg);background-position:left center;border-radius:2em;transition:background-position .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-switch .form-check-input{transition:none}}.form-switch .form-check-input:focus{--bs-form-switch-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3e%3ccircle r='3' fill='%2386b7fe'/%3e%3c/svg%3e")}.form-switch .form-check-input:checked{background-position:right center;--bs-form-switch-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3e%3ccircle r='3' fill='%23fff'/%3e%3c/svg%3e")}.form-switch.form-check-reverse{padding-right:2.5em;padding-left:0}.form-switch.form-check-reverse .form-check-input{margin-right:-2.5em;margin-left:0}.form-check-inline{display:inline-block;margin-right:1rem}.btn-check{position:absolute;clip:rect(0,0,0,0);pointer-events:none}.btn-check:disabled+.btn,.btn-check[disabled]+.btn{pointer-events:none;filter:none;opacity:.65}[data-bs-theme=dark] .form-switch .form-check-input:not(:checked):not(:focus){--bs-form-switch-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='-4 -4 8 8'%3e%3ccircle r='3' fill='rgba%28255, 255, 255, 0.25%29'/%3e%3c/svg%3e")}.form-range{width:100%;height:1.5rem;padding:0;-webkit-appearance:none;-moz-appearance:none;appearance:none;background-color:transparent}.form-range:focus{outline:0}.form-range:focus::-webkit-slider-thumb{box-shadow:0 0 0 1px #fff,0 0 0 .25rem rgba(13,110,253,.25)}.form-range:focus::-moz-range-thumb{box-shadow:0 0 0 1px #fff,0 0 0 .25rem rgba(13,110,253,.25)}.form-range::-moz-focus-outer{border:0}.form-range::-webkit-slider-thumb{width:1rem;height:1rem;margin-top:-.25rem;-webkit-appearance:none;appearance:none;background-color:#0d6efd;border:0;border-radius:1rem;-webkit-transition:background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;transition:background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-range::-webkit-slider-thumb{-webkit-transition:none;transition:none}}.form-range::-webkit-slider-thumb:active{background-color:#b6d4fe}.form-range::-webkit-slider-runnable-track{width:100%;height:.5rem;color:transparent;cursor:pointer;background-color:var(--bs-secondary-bg);border-color:transparent;border-radius:1rem}.form-range::-moz-range-thumb{width:1rem;height:1rem;-moz-appearance:none;appearance:none;background-color:#0d6efd;border:0;border-radius:1rem;-moz-transition:background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;transition:background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-range::-moz-range-thumb{-moz-transition:none;transition:none}}.form-range::-moz-range-thumb:active{background-color:#b6d4fe}.form-range::-moz-range-track{width:100%;height:.5rem;color:transparent;cursor:pointer;background-color:var(--bs-secondary-bg);border-color:transparent;border-radius:1rem}.form-range:disabled{pointer-events:none}.form-range:disabled::-webkit-slider-thumb{background-color:var(--bs-secondary-color)}.form-range:disabled::-moz-range-thumb{background-color:var(--bs-secondary-color)}.form-floating{position:relative}.form-floating>.form-control,.form-floating>.form-control-plaintext,.form-floating>.form-select{height:calc(3.5rem + calc(var(--bs-border-width) * 2));min-height:calc(3.5rem + calc(var(--bs-border-width) * 2));line-height:1.25}.form-floating>label{position:absolute;top:0;left:0;z-index:2;height:100%;padding:1rem .75rem;overflow:hidden;text-align:start;text-overflow:ellipsis;white-space:nowrap;pointer-events:none;border:var(--bs-border-width) solid transparent;transform-origin:0 0;transition:opacity .1s ease-in-out,transform .1s ease-in-out}@media (prefers-reduced-motion:reduce){.form-floating>label{transition:none}}.form-floating>.form-control,.form-floating>.form-control-plaintext{padding:1rem .75rem}.form-floating>.form-control-plaintext::-moz-placeholder,.form-floating>.form-control::-moz-placeholder{color:transparent}.form-floating>.form-control-plaintext::placeholder,.form-floating>.form-control::placeholder{color:transparent}.form-floating>.form-control-plaintext:not(:-moz-placeholder-shown),.form-floating>.form-control:not(:-moz-placeholder-shown){padding-top:1.625rem;padding-bottom:.625rem}.form-floating>.form-control-plaintext:focus,.form-floating>.form-control-plaintext:not(:placeholder-shown),.form-floating>.form-control:focus,.form-floating>.form-control:not(:placeholder-shown){padding-top:1.625rem;padding-bottom:.625rem}.form-floating>.form-control-plaintext:-webkit-autofill,.form-floating>.form-control:-webkit-autofill{padding-top:1.625rem;padding-bottom:.625rem}.form-floating>.form-select{padding-top:1.625rem;padding-bottom:.625rem}.form-floating>.form-control:not(:-moz-placeholder-shown)~label{color:rgba(var(--bs-body-color-rgb),.65);transform:scale(.85) translateY(-.5rem) translateX(.15rem)}.form-floating>.form-control-plaintext~label,.form-floating>.form-control:focus~label,.form-floating>.form-control:not(:placeholder-shown)~label,.form-floating>.form-select~label{color:rgba(var(--bs-body-color-rgb),.65);transform:scale(.85) translateY(-.5rem) translateX(.15rem)}.form-floating>.form-control:not(:-moz-placeholder-shown)~label::after{position:absolute;inset:1rem 0.375rem;z-index:-1;height:1.5em;content:"";background-color:var(--bs-body-bg);border-radius:var(--bs-border-radius)}.form-floating>.form-control-plaintext~label::after,.form-floating>.form-control:focus~label::after,.form-floating>.form-control:not(:placeholder-shown)~label::after,.form-floating>.form-select~label::after{position:absolute;inset:1rem 0.375rem;z-index:-1;height:1.5em;content:"";background-color:var(--bs-body-bg);border-radius:var(--bs-border-radius)}.form-floating>.form-control:-webkit-autofill~label{color:rgba(var(--bs-body-color-rgb),.65);transform:scale(.85) translateY(-.5rem) translateX(.15rem)}.form-floating>.form-control-plaintext~label{border-width:var(--bs-border-width) 0}.form-floating>.form-control:disabled~label,.form-floating>:disabled~label{color:#6c757d}.form-floating>.form-control:disabled~label::after,.form-floating>:disabled~label::after{background-color:var(--bs-secondary-bg)}.input-group{position:relative;display:flex;flex-wrap:wrap;align-items:stretch;width:100%}.input-group>.form-control,.input-group>.form-floating,.input-group>.form-select{position:relative;flex:1 1 auto;width:1%;min-width:0}.input-group>.form-control:focus,.input-group>.form-floating:focus-within,.input-group>.form-select:focus{z-index:5}.input-group .btn{position:relative;z-index:2}.input-group .btn:focus{z-index:5}.input-group-text{display:flex;align-items:center;padding:.375rem .75rem;font-size:1rem;font-weight:400;line-height:1.5;color:var(--bs-body-color);text-align:center;white-space:nowrap;background-color:var(--bs-tertiary-bg);border:var(--bs-border-width) solid var(--bs-border-color);border-radius:var(--bs-border-radius)}.input-group-lg>.btn,.input-group-lg>.form-control,.input-group-lg>.form-select,.input-group-lg>.input-group-text{padding:.5rem 1rem;font-size:1.25rem;border-radius:var(--bs-border-radius-lg)}.input-group-sm>.btn,.input-group-sm>.form-control,.input-group-sm>.form-select,.input-group-sm>.input-group-text{padding:.25rem .5rem;font-size:.875rem;border-radius:var(--bs-border-radius-sm)}.input-group-lg>.form-select,.input-group-sm>.form-select{padding-right:3rem}.input-group:not(.has-validation)>.dropdown-toggle:nth-last-child(n+3),.input-group:not(.has-validation)>.form-floating:not(:last-child)>.form-control,.input-group:not(.has-validation)>.form-floating:not(:last-child)>.form-select,.input-group:not(.has-validation)>:not(:last-child):not(.dropdown-toggle):not(.dropdown-menu):not(.form-floating){border-top-right-radius:0;border-bottom-right-radius:0}.input-group.has-validation>.dropdown-toggle:nth-last-child(n+4),.input-group.has-validation>.form-floating:nth-last-child(n+3)>.form-control,.input-group.has-validation>.form-floating:nth-last-child(n+3)>.form-select,.input-group.has-validation>:nth-last-child(n+3):not(.dropdown-toggle):not(.dropdown-menu):not(.form-floating){border-top-right-radius:0;border-bottom-right-radius:0}.input-group>:not(:first-child):not(.dropdown-menu):not(.valid-tooltip):not(.valid-feedback):not(.invalid-tooltip):not(.invalid-feedback){margin-left:calc(var(--bs-border-width) * -1);border-top-left-radius:0;border-bottom-left-radius:0}.input-group>.form-floating:not(:first-child)>.form-control,.input-group>.form-floating:not(:first-child)>.form-select{border-top-left-radius:0;border-bottom-left-radius:0}.valid-feedback{display:none;width:100%;margin-top:.25rem;font-size:.875em;color:var(--bs-form-valid-color)}.valid-tooltip{position:absolute;top:100%;z-index:5;display:none;max-width:100%;padding:.25rem .5rem;margin-top:.1rem;font-size:.875rem;color:#fff;background-color:var(--bs-success);border-radius:var(--bs-border-radius)}.is-valid~.valid-feedback,.is-valid~.valid-tooltip,.was-validated :valid~.valid-feedback,.was-validated :valid~.valid-tooltip{display:block}.form-control.is-valid,.was-validated .form-control:valid{border-color:var(--bs-form-valid-border-color);padding-right:calc(1.5em + .75rem);background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3e%3cpath fill='%23198754' d='M2.3 6.73.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3e%3c/svg%3e");background-repeat:no-repeat;background-position:right calc(.375em + .1875rem) center;background-size:calc(.75em + .375rem) calc(.75em + .375rem)}.form-control.is-valid:focus,.was-validated .form-control:valid:focus{border-color:var(--bs-form-valid-border-color);box-shadow:0 0 0 .25rem rgba(var(--bs-success-rgb),.25)}.was-validated textarea.form-control:valid,textarea.form-control.is-valid{padding-right:calc(1.5em + .75rem);background-position:top calc(.375em + .1875rem) right calc(.375em + .1875rem)}.form-select.is-valid,.was-validated .form-select:valid{border-color:var(--bs-form-valid-border-color)}.form-select.is-valid:not([multiple]):not([size]),.form-select.is-valid:not([multiple])[size="1"],.was-validated .form-select:valid:not([multiple]):not([size]),.was-validated .form-select:valid:not([multiple])[size="1"]{--bs-form-select-bg-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 8 8'%3e%3cpath fill='%23198754' d='M2.3 6.73.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3e%3c/svg%3e");padding-right:4.125rem;background-position:right .75rem center,center right 2.25rem;background-size:16px 12px,calc(.75em + .375rem) calc(.75em + .375rem)}.form-select.is-valid:focus,.was-validated .form-select:valid:focus{border-color:var(--bs-form-valid-border-color);box-shadow:0 0 0 .25rem rgba(var(--bs-success-rgb),.25)}.form-control-color.is-valid,.was-validated .form-control-color:valid{width:calc(3rem + calc(1.5em + .75rem))}.form-check-input.is-valid,.was-validated .form-check-input:valid{border-color:var(--bs-form-valid-border-color)}.form-check-input.is-valid:checked,.was-validated .form-check-input:valid:checked{background-color:var(--bs-form-valid-color)}.form-check-input.is-valid:focus,.was-validated .form-check-input:valid:focus{box-shadow:0 0 0 .25rem rgba(var(--bs-success-rgb),.25)}.form-check-input.is-valid~.form-check-label,.was-validated .form-check-input:valid~.form-check-label{color:var(--bs-form-valid-color)}.form-check-inline .form-check-input~.valid-feedback{margin-left:.5em}.input-group>.form-control:not(:focus).is-valid,.input-group>.form-floating:not(:focus-within).is-valid,.input-group>.form-select:not(:focus).is-valid,.was-validated .input-group>.form-control:not(:focus):valid,.was-validated .input-group>.form-floating:not(:focus-within):valid,.was-validated .input-group>.form-select:not(:focus):valid{z-index:3}.invalid-feedback{display:none;width:100%;margin-top:.25rem;font-size:.875em;color:var(--bs-form-invalid-color)}.invalid-tooltip{position:absolute;top:100%;z-index:5;display:none;max-width:100%;padding:.25rem .5rem;margin-top:.1rem;font-size:.875rem;color:#fff;background-color:var(--bs-danger);border-radius:var(--bs-border-radius)}.is-invalid~.invalid-feedback,.is-invalid~.invalid-tooltip,.was-validated :invalid~.invalid-feedback,.was-validated :invalid~.invalid-tooltip{display:block}.form-control.is-invalid,.was-validated .form-control:invalid{border-color:var(--bs-form-invalid-border-color);padding-right:calc(1.5em + .75rem);background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 12 12' width='12' height='12' fill='none' stroke='%23dc3545'%3e%3ccircle cx='6' cy='6' r='4.5'/%3e%3cpath stroke-linejoin='round' d='M5.8 3.6h.4L6 6.5z'/%3e%3ccircle cx='6' cy='8.2' r='.6' fill='%23dc3545' stroke='none'/%3e%3c/svg%3e");background-repeat:no-repeat;background-position:right calc(.375em + .1875rem) center;background-size:calc(.75em + .375rem) calc(.75em + .375rem)}.form-control.is-invalid:focus,.was-validated .form-control:invalid:focus{border-color:var(--bs-form-invalid-border-color);box-shadow:0 0 0 .25rem rgba(var(--bs-danger-rgb),.25)}.was-validated textarea.form-control:invalid,textarea.form-control.is-invalid{padding-right:calc(1.5em + .75rem);background-position:top calc(.375em + .1875rem) right calc(.375em + .1875rem)}.form-select.is-invalid,.was-validated .form-select:invalid{border-color:var(--bs-form-invalid-border-color)}.form-select.is-invalid:not([multiple]):not([size]),.form-select.is-invalid:not([multiple])[size="1"],.was-validated .form-select:invalid:not([multiple]):not([size]),.was-validated .form-select:invalid:not([multiple])[size="1"]{--bs-form-select-bg-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 12 12' width='12' height='12' fill='none' stroke='%23dc3545'%3e%3ccircle cx='6' cy='6' r='4.5'/%3e%3cpath stroke-linejoin='round' d='M5.8 3.6h.4L6 6.5z'/%3e%3ccircle cx='6' cy='8.2' r='.6' fill='%23dc3545' stroke='none'/%3e%3c/svg%3e");padding-right:4.125rem;background-position:right .75rem center,center right 2.25rem;background-size:16px 12px,calc(.75em + .375rem) calc(.75em + .375rem)}.form-select.is-invalid:focus,.was-validated .form-select:invalid:focus{border-color:var(--bs-form-invalid-border-color);box-shadow:0 0 0 .25rem rgba(var(--bs-danger-rgb),.25)}.form-control-color.is-invalid,.was-validated .form-control-color:invalid{width:calc(3rem + calc(1.5em + .75rem))}.form-check-input.is-invalid,.was-validated .form-check-input:invalid{border-color:var(--bs-form-invalid-border-color)}.form-check-input.is-invalid:checked,.was-validated .form-check-input:invalid:checked{background-color:var(--bs-form-invalid-color)}.form-check-input.is-invalid:focus,.was-validated .form-check-input:invalid:focus{box-shadow:0 0 0 .25rem rgba(var(--bs-danger-rgb),.25)}.form-check-input.is-invalid~.form-check-label,.was-validated .form-check-input:invalid~.form-check-label{color:var(--bs-form-invalid-color)}.form-check-inline .form-check-input~.invalid-feedback{margin-left:.5em}.input-group>.form-control:not(:focus).is-invalid,.input-group>.form-floating:not(:focus-within).is-invalid,.input-group>.form-select:not(:focus).is-invalid,.was-validated .input-group>.form-control:not(:focus):invalid,.was-validated .input-group>.form-floating:not(:focus-within):invalid,.was-validated .input-group>.form-select:not(:focus):invalid{z-index:4}.btn{--bs-btn-padding-x:0.75rem;--bs-btn-padding-y:0.375rem;--bs-btn-font-family: ;--bs-btn-font-size:1rem;--bs-btn-font-weight:400;--bs-btn-line-height:1.5;--bs-btn-color:var(--bs-body-color);--bs-btn-bg:transparent;--bs-btn-border-width:var(--bs-border-width);--bs-btn-border-color:transparent;--bs-btn-border-radius:var(--bs-border-radius);--bs-btn-hover-border-color:transparent;--bs-btn-box-shadow:inset 0 1px 0 rgba(255, 255, 255, 0.15),0 1px 1px rgba(0, 0, 0, 0.075);--bs-btn-disabled-opacity:0.65;--bs-btn-focus-box-shadow:0 0 0 0.25rem rgba(var(--bs-btn-focus-shadow-rgb), .5);display:inline-block;padding:var(--bs-btn-padding-y) var(--bs-btn-padding-x);font-family:var(--bs-btn-font-family);font-size:var(--bs-btn-font-size);font-weight:var(--bs-btn-font-weight);line-height:var(--bs-btn-line-height);color:var(--bs-btn-color);text-align:center;text-decoration:none;vertical-align:middle;cursor:pointer;-webkit-user-select:none;-moz-user-select:none;user-select:none;border:var(--bs-btn-border-width) solid var(--bs-btn-border-color);border-radius:var(--bs-btn-border-radius);background-color:var(--bs-btn-bg);transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.btn{transition:none}}.btn:hover{color:var(--bs-btn-hover-color);background-color:var(--bs-btn-hover-bg);border-color:var(--bs-btn-hover-border-color)}.btn-check+.btn:hover{color:var(--bs-btn-color);background-color:var(--bs-btn-bg);border-color:var(--bs-btn-border-color)}.btn:focus-visible{color:var(--bs-btn-hover-color);background-color:var(--bs-btn-hover-bg);border-color:var(--bs-btn-hover-border-color);outline:0;box-shadow:var(--bs-btn-focus-box-shadow)}.btn-check:focus-visible+.btn{border-color:var(--bs-btn-hover-border-color);outline:0;box-shadow:var(--bs-btn-focus-box-shadow)}.btn-check:checked+.btn,.btn.active,.btn.show,.btn:first-child:active,:not(.btn-check)+.btn:active{color:var(--bs-btn-active-color);background-color:var(--bs-btn-active-bg);border-color:var(--bs-btn-active-border-color)}.btn-check:checked+.btn:focus-visible,.btn.active:focus-visible,.btn.show:focus-visible,.btn:first-child:active:focus-visible,:not(.btn-check)+.btn:active:focus-visible{box-shadow:var(--bs-btn-focus-box-shadow)}.btn-check:checked:focus-visible+.btn{box-shadow:var(--bs-btn-focus-box-shadow)}.btn.disabled,.btn:disabled,fieldset:disabled .btn{color:var(--bs-btn-disabled-color);pointer-events:none;background-color:var(--bs-btn-disabled-bg);border-color:var(--bs-btn-disabled-border-color);opacity:var(--bs-btn-disabled-opacity)}.btn-primary{--bs-btn-color:#fff;--bs-btn-bg:#0d6efd;--bs-btn-border-color:#0d6efd;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#0b5ed7;--bs-btn-hover-border-color:#0a58ca;--bs-btn-focus-shadow-rgb:49,132,253;--bs-btn-active-color:#fff;--bs-btn-active-bg:#0a58ca;--bs-btn-active-border-color:#0a53be;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#fff;--bs-btn-disabled-bg:#0d6efd;--bs-btn-disabled-border-color:#0d6efd}.btn-secondary{--bs-btn-color:#fff;--bs-btn-bg:#6c757d;--bs-btn-border-color:#6c757d;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#5c636a;--bs-btn-hover-border-color:#565e64;--bs-btn-focus-shadow-rgb:130,138,145;--bs-btn-active-color:#fff;--bs-btn-active-bg:#565e64;--bs-btn-active-border-color:#51585e;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#fff;--bs-btn-disabled-bg:#6c757d;--bs-btn-disabled-border-color:#6c757d}.btn-success{--bs-btn-color:#fff;--bs-btn-bg:#198754;--bs-btn-border-color:#198754;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#157347;--bs-btn-hover-border-color:#146c43;--bs-btn-focus-shadow-rgb:60,153,110;--bs-btn-active-color:#fff;--bs-btn-active-bg:#146c43;--bs-btn-active-border-color:#13653f;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#fff;--bs-btn-disabled-bg:#198754;--bs-btn-disabled-border-color:#198754}.btn-info{--bs-btn-color:#000;--bs-btn-bg:#0dcaf0;--bs-btn-border-color:#0dcaf0;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#31d2f2;--bs-btn-hover-border-color:#25cff2;--bs-btn-focus-shadow-rgb:11,172,204;--bs-btn-active-color:#000;--bs-btn-active-bg:#3dd5f3;--bs-btn-active-border-color:#25cff2;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#000;--bs-btn-disabled-bg:#0dcaf0;--bs-btn-disabled-border-color:#0dcaf0}.btn-warning{--bs-btn-color:#000;--bs-btn-bg:#ffc107;--bs-btn-border-color:#ffc107;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#ffca2c;--bs-btn-hover-border-color:#ffc720;--bs-btn-focus-shadow-rgb:217,164,6;--bs-btn-active-color:#000;--bs-btn-active-bg:#ffcd39;--bs-btn-active-border-color:#ffc720;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#000;--bs-btn-disabled-bg:#ffc107;--bs-btn-disabled-border-color:#ffc107}.btn-danger{--bs-btn-color:#fff;--bs-btn-bg:#dc3545;--bs-btn-border-color:#dc3545;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#bb2d3b;--bs-btn-hover-border-color:#b02a37;--bs-btn-focus-shadow-rgb:225,83,97;--bs-btn-active-color:#fff;--bs-btn-active-bg:#b02a37;--bs-btn-active-border-color:#a52834;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#fff;--bs-btn-disabled-bg:#dc3545;--bs-btn-disabled-border-color:#dc3545}.btn-light{--bs-btn-color:#000;--bs-btn-bg:#f8f9fa;--bs-btn-border-color:#f8f9fa;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#d3d4d5;--bs-btn-hover-border-color:#c6c7c8;--bs-btn-focus-shadow-rgb:211,212,213;--bs-btn-active-color:#000;--bs-btn-active-bg:#c6c7c8;--bs-btn-active-border-color:#babbbc;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#000;--bs-btn-disabled-bg:#f8f9fa;--bs-btn-disabled-border-color:#f8f9fa}.btn-dark{--bs-btn-color:#fff;--bs-btn-bg:#212529;--bs-btn-border-color:#212529;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#424649;--bs-btn-hover-border-color:#373b3e;--bs-btn-focus-shadow-rgb:66,70,73;--bs-btn-active-color:#fff;--bs-btn-active-bg:#4d5154;--bs-btn-active-border-color:#373b3e;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#fff;--bs-btn-disabled-bg:#212529;--bs-btn-disabled-border-color:#212529}.btn-outline-primary{--bs-btn-color:#0d6efd;--bs-btn-border-color:#0d6efd;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#0d6efd;--bs-btn-hover-border-color:#0d6efd;--bs-btn-focus-shadow-rgb:13,110,253;--bs-btn-active-color:#fff;--bs-btn-active-bg:#0d6efd;--bs-btn-active-border-color:#0d6efd;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#0d6efd;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#0d6efd;--bs-gradient:none}.btn-outline-secondary{--bs-btn-color:#6c757d;--bs-btn-border-color:#6c757d;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#6c757d;--bs-btn-hover-border-color:#6c757d;--bs-btn-focus-shadow-rgb:108,117,125;--bs-btn-active-color:#fff;--bs-btn-active-bg:#6c757d;--bs-btn-active-border-color:#6c757d;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#6c757d;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#6c757d;--bs-gradient:none}.btn-outline-success{--bs-btn-color:#198754;--bs-btn-border-color:#198754;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#198754;--bs-btn-hover-border-color:#198754;--bs-btn-focus-shadow-rgb:25,135,84;--bs-btn-active-color:#fff;--bs-btn-active-bg:#198754;--bs-btn-active-border-color:#198754;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#198754;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#198754;--bs-gradient:none}.btn-outline-info{--bs-btn-color:#0dcaf0;--bs-btn-border-color:#0dcaf0;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#0dcaf0;--bs-btn-hover-border-color:#0dcaf0;--bs-btn-focus-shadow-rgb:13,202,240;--bs-btn-active-color:#000;--bs-btn-active-bg:#0dcaf0;--bs-btn-active-border-color:#0dcaf0;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#0dcaf0;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#0dcaf0;--bs-gradient:none}.btn-outline-warning{--bs-btn-color:#ffc107;--bs-btn-border-color:#ffc107;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#ffc107;--bs-btn-hover-border-color:#ffc107;--bs-btn-focus-shadow-rgb:255,193,7;--bs-btn-active-color:#000;--bs-btn-active-bg:#ffc107;--bs-btn-active-border-color:#ffc107;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#ffc107;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#ffc107;--bs-gradient:none}.btn-outline-danger{--bs-btn-color:#dc3545;--bs-btn-border-color:#dc3545;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#dc3545;--bs-btn-hover-border-color:#dc3545;--bs-btn-focus-shadow-rgb:220,53,69;--bs-btn-active-color:#fff;--bs-btn-active-bg:#dc3545;--bs-btn-active-border-color:#dc3545;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#dc3545;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#dc3545;--bs-gradient:none}.btn-outline-light{--bs-btn-color:#f8f9fa;--bs-btn-border-color:#f8f9fa;--bs-btn-hover-color:#000;--bs-btn-hover-bg:#f8f9fa;--bs-btn-hover-border-color:#f8f9fa;--bs-btn-focus-shadow-rgb:248,249,250;--bs-btn-active-color:#000;--bs-btn-active-bg:#f8f9fa;--bs-btn-active-border-color:#f8f9fa;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#f8f9fa;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#f8f9fa;--bs-gradient:none}.btn-outline-dark{--bs-btn-color:#212529;--bs-btn-border-color:#212529;--bs-btn-hover-color:#fff;--bs-btn-hover-bg:#212529;--bs-btn-hover-border-color:#212529;--bs-btn-focus-shadow-rgb:33,37,41;--bs-btn-active-color:#fff;--bs-btn-active-bg:#212529;--bs-btn-active-border-color:#212529;--bs-btn-active-shadow:inset 0 3px 5px rgba(0, 0, 0, 0.125);--bs-btn-disabled-color:#212529;--bs-btn-disabled-bg:transparent;--bs-btn-disabled-border-color:#212529;--bs-gradient:none}.btn-link{--bs-btn-font-weight:400;--bs-btn-color:var(--bs-link-color);--bs-btn-bg:transparent;--bs-btn-border-color:transparent;--bs-btn-hover-color:var(--bs-link-hover-color);--bs-btn-hover-border-color:transparent;--bs-btn-active-color:var(--bs-link-hover-color);--bs-btn-active-border-color:transparent;--bs-btn-disabled-color:#6c757d;--bs-btn-disabled-border-color:transparent;--bs-btn-box-shadow:0 0 0 #000;--bs-btn-focus-shadow-rgb:49,132,253;text-decoration:underline}.btn-link:focus-visible{color:var(--bs-btn-color)}.btn-link:hover{color:var(--bs-btn-hover-color)}.btn-group-lg>.btn,.btn-lg{--bs-btn-padding-y:0.5rem;--bs-btn-padding-x:1rem;--bs-btn-font-size:1.25rem;--bs-btn-border-radius:var(--bs-border-radius-lg)}.btn-group-sm>.btn,.btn-sm{--bs-btn-padding-y:0.25rem;--bs-btn-padding-x:0.5rem;--bs-btn-font-size:0.875rem;--bs-btn-border-radius:var(--bs-border-radius-sm)}.fade{transition:opacity .15s linear}@media (prefers-reduced-motion:reduce){.fade{transition:none}}.fade:not(.show){opacity:0}.collapse:not(.show){display:none}.collapsing{height:0;overflow:hidden;transition:height .35s ease}@media (prefers-reduced-motion:reduce){.collapsing{transition:none}}.collapsing.collapse-horizontal{width:0;height:auto;transition:width .35s ease}@media (prefers-reduced-motion:reduce){.collapsing.collapse-horizontal{transition:none}}.dropdown,.dropdown-center,.dropend,.dropstart,.dropup,.dropup-center{position:relative}.dropdown-toggle{white-space:nowrap}.dropdown-toggle::after{display:inline-block;margin-left:.255em;vertical-align:.255em;content:"";border-top:.3em solid;border-right:.3em solid transparent;border-bottom:0;border-left:.3em solid transparent}.dropdown-toggle:empty::after{margin-left:0}.dropdown-menu{--bs-dropdown-zindex:1000;--bs-dropdown-min-width:10rem;--bs-dropdown-padding-x:0;--bs-dropdown-padding-y:0.5rem;--bs-dropdown-spacer:0.125rem;--bs-dropdown-font-size:1rem;--bs-dropdown-color:var(--bs-body-color);--bs-dropdown-bg:var(--bs-body-bg);--bs-dropdown-border-color:var(--bs-border-color-translucent);--bs-dropdown-border-radius:var(--bs-border-radius);--bs-dropdown-border-width:var(--bs-border-width);--bs-dropdown-inner-border-radius:calc(var(--bs-border-radius) - var(--bs-border-width));--bs-dropdown-divider-bg:var(--bs-border-color-translucent);--bs-dropdown-divider-margin-y:0.5rem;--bs-dropdown-box-shadow:var(--bs-box-shadow);--bs-dropdown-link-color:var(--bs-body-color);--bs-dropdown-link-hover-color:var(--bs-body-color);--bs-dropdown-link-hover-bg:var(--bs-tertiary-bg);--bs-dropdown-link-active-color:#fff;--bs-dropdown-link-active-bg:#0d6efd;--bs-dropdown-link-disabled-color:var(--bs-tertiary-color);--bs-dropdown-item-padding-x:1rem;--bs-dropdown-item-padding-y:0.25rem;--bs-dropdown-header-color:#6c757d;--bs-dropdown-header-padding-x:1rem;--bs-dropdown-header-padding-y:0.5rem;position:absolute;z-index:var(--bs-dropdown-zindex);display:none;min-width:var(--bs-dropdown-min-width);padding:var(--bs-dropdown-padding-y) var(--bs-dropdown-padding-x);margin:0;font-size:var(--bs-dropdown-font-size);color:var(--bs-dropdown-color);text-align:left;list-style:none;background-color:var(--bs-dropdown-bg);background-clip:padding-box;border:var(--bs-dropdown-border-width) solid var(--bs-dropdown-border-color);border-radius:var(--bs-dropdown-border-radius)}.dropdown-menu[data-bs-popper]{top:100%;left:0;margin-top:var(--bs-dropdown-spacer)}.dropdown-menu-start{--bs-position:start}.dropdown-menu-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-end{--bs-position:end}.dropdown-menu-end[data-bs-popper]{right:0;left:auto}@media (min-width:576px){.dropdown-menu-sm-start{--bs-position:start}.dropdown-menu-sm-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-sm-end{--bs-position:end}.dropdown-menu-sm-end[data-bs-popper]{right:0;left:auto}}@media (min-width:768px){.dropdown-menu-md-start{--bs-position:start}.dropdown-menu-md-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-md-end{--bs-position:end}.dropdown-menu-md-end[data-bs-popper]{right:0;left:auto}}@media (min-width:992px){.dropdown-menu-lg-start{--bs-position:start}.dropdown-menu-lg-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-lg-end{--bs-position:end}.dropdown-menu-lg-end[data-bs-popper]{right:0;left:auto}}@media (min-width:1200px){.dropdown-menu-xl-start{--bs-position:start}.dropdown-menu-xl-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-xl-end{--bs-position:end}.dropdown-menu-xl-end[data-bs-popper]{right:0;left:auto}}@media (min-width:1400px){.dropdown-menu-xxl-start{--bs-position:start}.dropdown-menu-xxl-start[data-bs-popper]{right:auto;left:0}.dropdown-menu-xxl-end{--bs-position:end}.dropdown-menu-xxl-end[data-bs-popper]{right:0;left:auto}}.dropup .dropdown-menu[data-bs-popper]{top:auto;bottom:100%;margin-top:0;margin-bottom:var(--bs-dropdown-spacer)}.dropup .dropdown-toggle::after{display:inline-block;margin-left:.255em;vertical-align:.255em;content:"";border-top:0;border-right:.3em solid transparent;border-bottom:.3em solid;border-left:.3em solid transparent}.dropup .dropdown-toggle:empty::after{margin-left:0}.dropend .dropdown-menu[data-bs-popper]{top:0;right:auto;left:100%;margin-top:0;margin-left:var(--bs-dropdown-spacer)}.dropend .dropdown-toggle::after{display:inline-block;margin-left:.255em;vertical-align:.255em;content:"";border-top:.3em solid transparent;border-right:0;border-bottom:.3em solid transparent;border-left:.3em solid}.dropend .dropdown-toggle:empty::after{margin-left:0}.dropend .dropdown-toggle::after{vertical-align:0}.dropstart .dropdown-menu[data-bs-popper]{top:0;right:100%;left:auto;margin-top:0;margin-right:var(--bs-dropdown-spacer)}.dropstart .dropdown-toggle::after{display:inline-block;margin-left:.255em;vertical-align:.255em;content:""}.dropstart .dropdown-toggle::after{display:none}.dropstart .dropdown-toggle::before{display:inline-block;margin-right:.255em;vertical-align:.255em;content:"";border-top:.3em solid transparent;border-right:.3em solid;border-bottom:.3em solid transparent}.dropstart .dropdown-toggle:empty::after{margin-left:0}.dropstart .dropdown-toggle::before{vertical-align:0}.dropdown-divider{height:0;margin:var(--bs-dropdown-divider-margin-y) 0;overflow:hidden;border-top:1px solid var(--bs-dropdown-divider-bg);opacity:1}.dropdown-item{display:block;width:100%;padding:var(--bs-dropdown-item-padding-y) var(--bs-dropdown-item-padding-x);clear:both;font-weight:400;color:var(--bs-dropdown-link-color);text-align:inherit;text-decoration:none;white-space:nowrap;background-color:transparent;border:0;border-radius:var(--bs-dropdown-item-border-radius,0)}.dropdown-item:focus,.dropdown-item:hover{color:var(--bs-dropdown-link-hover-color);background-color:var(--bs-dropdown-link-hover-bg)}.dropdown-item.active,.dropdown-item:active{color:var(--bs-dropdown-link-active-color);text-decoration:none;background-color:var(--bs-dropdown-link-active-bg)}.dropdown-item.disabled,.dropdown-item:disabled{color:var(--bs-dropdown-link-disabled-color);pointer-events:none;background-color:transparent}.dropdown-menu.show{display:block}.dropdown-header{display:block;padding:var(--bs-dropdown-header-padding-y) var(--bs-dropdown-header-padding-x);margin-bottom:0;font-size:.875rem;color:var(--bs-dropdown-header-color);white-space:nowrap}.dropdown-item-text{display:block;padding:var(--bs-dropdown-item-padding-y) var(--bs-dropdown-item-padding-x);color:var(--bs-dropdown-link-color)}.dropdown-menu-dark{--bs-dropdown-color:#dee2e6;--bs-dropdown-bg:#343a40;--bs-dropdown-border-color:var(--bs-border-color-translucent);--bs-dropdown-box-shadow: ;--bs-dropdown-link-color:#dee2e6;--bs-dropdown-link-hover-color:#fff;--bs-dropdown-divider-bg:var(--bs-border-color-translucent);--bs-dropdown-link-hover-bg:rgba(255, 255, 255, 0.15);--bs-dropdown-link-active-color:#fff;--bs-dropdown-link-active-bg:#0d6efd;--bs-dropdown-link-disabled-color:#adb5bd;--bs-dropdown-header-color:#adb5bd}.btn-group,.btn-group-vertical{position:relative;display:inline-flex;vertical-align:middle}.btn-group-vertical>.btn,.btn-group>.btn{position:relative;flex:1 1 auto}.btn-group-vertical>.btn-check:checked+.btn,.btn-group-vertical>.btn-check:focus+.btn,.btn-group-vertical>.btn.active,.btn-group-vertical>.btn:active,.btn-group-vertical>.btn:focus,.btn-group-vertical>.btn:hover,.btn-group>.btn-check:checked+.btn,.btn-group>.btn-check:focus+.btn,.btn-group>.btn.active,.btn-group>.btn:active,.btn-group>.btn:focus,.btn-group>.btn:hover{z-index:1}.btn-toolbar{display:flex;flex-wrap:wrap;justify-content:flex-start}.btn-toolbar .input-group{width:auto}.btn-group{border-radius:var(--bs-border-radius)}.btn-group>.btn-group:not(:first-child),.btn-group>:not(.btn-check:first-child)+.btn{margin-left:calc(var(--bs-border-width) * -1)}.btn-group>.btn-group:not(:last-child)>.btn,.btn-group>.btn.dropdown-toggle-split:first-child,.btn-group>.btn:not(:last-child):not(.dropdown-toggle){border-top-right-radius:0;border-bottom-right-radius:0}.btn-group>.btn-group:not(:first-child)>.btn,.btn-group>.btn:nth-child(n+3),.btn-group>:not(.btn-check)+.btn{border-top-left-radius:0;border-bottom-left-radius:0}.dropdown-toggle-split{padding-right:.5625rem;padding-left:.5625rem}.dropdown-toggle-split::after,.dropend .dropdown-toggle-split::after,.dropup .dropdown-toggle-split::after{margin-left:0}.dropstart .dropdown-toggle-split::before{margin-right:0}.btn-group-sm>.btn+.dropdown-toggle-split,.btn-sm+.dropdown-toggle-split{padding-right:.375rem;padding-left:.375rem}.btn-group-lg>.btn+.dropdown-toggle-split,.btn-lg+.dropdown-toggle-split{padding-right:.75rem;padding-left:.75rem}.btn-group-vertical{flex-direction:column;align-items:flex-start;justify-content:center}.btn-group-vertical>.btn,.btn-group-vertical>.btn-group{width:100%}.btn-group-vertical>.btn-group:not(:first-child),.btn-group-vertical>.btn:not(:first-child){margin-top:calc(var(--bs-border-width) * -1)}.btn-group-vertical>.btn-group:not(:last-child)>.btn,.btn-group-vertical>.btn:not(:last-child):not(.dropdown-toggle){border-bottom-right-radius:0;border-bottom-left-radius:0}.btn-group-vertical>.btn-group:not(:first-child)>.btn,.btn-group-vertical>.btn~.btn{border-top-left-radius:0;border-top-right-radius:0}.nav{--bs-nav-link-padding-x:1rem;--bs-nav-link-padding-y:0.5rem;--bs-nav-link-font-weight: ;--bs-nav-link-color:var(--bs-link-color);--bs-nav-link-hover-color:var(--bs-link-hover-color);--bs-nav-link-disabled-color:var(--bs-secondary-color);display:flex;flex-wrap:wrap;padding-left:0;margin-bottom:0;list-style:none}.nav-link{display:block;padding:var(--bs-nav-link-padding-y) var(--bs-nav-link-padding-x);font-size:var(--bs-nav-link-font-size);font-weight:var(--bs-nav-link-font-weight);color:var(--bs-nav-link-color);text-decoration:none;background:0 0;border:0;transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out}@media (prefers-reduced-motion:reduce){.nav-link{transition:none}}.nav-link:focus,.nav-link:hover{color:var(--bs-nav-link-hover-color)}.nav-link:focus-visible{outline:0;box-shadow:0 0 0 .25rem rgba(13,110,253,.25)}.nav-link.disabled,.nav-link:disabled{color:var(--bs-nav-link-disabled-color);pointer-events:none;cursor:default}.nav-tabs{--bs-nav-tabs-border-width:var(--bs-border-width);--bs-nav-tabs-border-color:var(--bs-border-color);--bs-nav-tabs-border-radius:var(--bs-border-radius);--bs-nav-tabs-link-hover-border-color:var(--bs-secondary-bg) var(--bs-secondary-bg) var(--bs-border-color);--bs-nav-tabs-link-active-color:var(--bs-emphasis-color);--bs-nav-tabs-link-active-bg:var(--bs-body-bg);--bs-nav-tabs-link-active-border-color:var(--bs-border-color) var(--bs-border-color) var(--bs-body-bg);border-bottom:var(--bs-nav-tabs-border-width) solid var(--bs-nav-tabs-border-color)}.nav-tabs .nav-link{margin-bottom:calc(-1 * var(--bs-nav-tabs-border-width));border:var(--bs-nav-tabs-border-width) solid transparent;border-top-left-radius:var(--bs-nav-tabs-border-radius);border-top-right-radius:var(--bs-nav-tabs-border-radius)}.nav-tabs .nav-link:focus,.nav-tabs .nav-link:hover{isolation:isolate;border-color:var(--bs-nav-tabs-link-hover-border-color)}.nav-tabs .nav-item.show .nav-link,.nav-tabs .nav-link.active{color:var(--bs-nav-tabs-link-active-color);background-color:var(--bs-nav-tabs-link-active-bg);border-color:var(--bs-nav-tabs-link-active-border-color)}.nav-tabs .dropdown-menu{margin-top:calc(-1 * var(--bs-nav-tabs-border-width));border-top-left-radius:0;border-top-right-radius:0}.nav-pills{--bs-nav-pills-border-radius:var(--bs-border-radius);--bs-nav-pills-link-active-color:#fff;--bs-nav-pills-link-active-bg:#0d6efd}.nav-pills .nav-link{border-radius:var(--bs-nav-pills-border-radius)}.nav-pills .nav-link.active,.nav-pills .show>.nav-link{color:var(--bs-nav-pills-link-active-color);background-color:var(--bs-nav-pills-link-active-bg)}.nav-underline{--bs-nav-underline-gap:1rem;--bs-nav-underline-border-width:0.125rem;--bs-nav-underline-link-active-color:var(--bs-emphasis-color);gap:var(--bs-nav-underline-gap)}.nav-underline .nav-link{padding-right:0;padding-left:0;border-bottom:var(--bs-nav-underline-border-width) solid transparent}.nav-underline .nav-link:focus,.nav-underline .nav-link:hover{border-bottom-color:currentcolor}.nav-underline .nav-link.active,.nav-underline .show>.nav-link{font-weight:700;color:var(--bs-nav-underline-link-active-color);border-bottom-color:currentcolor}.nav-fill .nav-item,.nav-fill>.nav-link{flex:1 1 auto;text-align:center}.nav-justified .nav-item,.nav-justified>.nav-link{flex-basis:0;flex-grow:1;text-align:center}.nav-fill .nav-item .nav-link,.nav-justified .nav-item .nav-link{width:100%}.tab-content>.tab-pane{display:none}.tab-content>.active{display:block}.navbar{--bs-navbar-padding-x:0;--bs-navbar-padding-y:0.5rem;--bs-navbar-color:rgba(var(--bs-emphasis-color-rgb), 0.65);--bs-navbar-hover-color:rgba(var(--bs-emphasis-color-rgb), 0.8);--bs-navbar-disabled-color:rgba(var(--bs-emphasis-color-rgb), 0.3);--bs-navbar-active-color:rgba(var(--bs-emphasis-color-rgb), 1);--bs-navbar-brand-padding-y:0.3125rem;--bs-navbar-brand-margin-end:1rem;--bs-navbar-brand-font-size:1.25rem;--bs-navbar-brand-color:rgba(var(--bs-emphasis-color-rgb), 1);--bs-navbar-brand-hover-color:rgba(var(--bs-emphasis-color-rgb), 1);--bs-navbar-nav-link-padding-x:0.5rem;--bs-navbar-toggler-padding-y:0.25rem;--bs-navbar-toggler-padding-x:0.75rem;--bs-navbar-toggler-font-size:1.25rem;--bs-navbar-toggler-icon-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 30 30'%3e%3cpath stroke='rgba%2833, 37, 41, 0.75%29' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/%3e%3c/svg%3e");--bs-navbar-toggler-border-color:rgba(var(--bs-emphasis-color-rgb), 0.15);--bs-navbar-toggler-border-radius:var(--bs-border-radius);--bs-navbar-toggler-focus-width:0.25rem;--bs-navbar-toggler-transition:box-shadow 0.15s ease-in-out;position:relative;display:flex;flex-wrap:wrap;align-items:center;justify-content:space-between;padding:var(--bs-navbar-padding-y) var(--bs-navbar-padding-x)}.navbar>.container,.navbar>.container-fluid,.navbar>.container-lg,.navbar>.container-md,.navbar>.container-sm,.navbar>.container-xl,.navbar>.container-xxl{display:flex;flex-wrap:inherit;align-items:center;justify-content:space-between}.navbar-brand{padding-top:var(--bs-navbar-brand-padding-y);padding-bottom:var(--bs-navbar-brand-padding-y);margin-right:var(--bs-navbar-brand-margin-end);font-size:var(--bs-navbar-brand-font-size);color:var(--bs-navbar-brand-color);text-decoration:none;white-space:nowrap}.navbar-brand:focus,.navbar-brand:hover{color:var(--bs-navbar-brand-hover-color)}.navbar-nav{--bs-nav-link-padding-x:0;--bs-nav-link-padding-y:0.5rem;--bs-nav-link-font-weight: ;--bs-nav-link-color:var(--bs-navbar-color);--bs-nav-link-hover-color:var(--bs-navbar-hover-color);--bs-nav-link-disabled-color:var(--bs-navbar-disabled-color);display:flex;flex-direction:column;padding-left:0;margin-bottom:0;list-style:none}.navbar-nav .nav-link.active,.navbar-nav .nav-link.show{color:var(--bs-navbar-active-color)}.navbar-nav .dropdown-menu{position:static}.navbar-text{padding-top:.5rem;padding-bottom:.5rem;color:var(--bs-navbar-color)}.navbar-text a,.navbar-text a:focus,.navbar-text a:hover{color:var(--bs-navbar-active-color)}.navbar-collapse{flex-basis:100%;flex-grow:1;align-items:center}.navbar-toggler{padding:var(--bs-navbar-toggler-padding-y) var(--bs-navbar-toggler-padding-x);font-size:var(--bs-navbar-toggler-font-size);line-height:1;color:var(--bs-navbar-color);background-color:transparent;border:var(--bs-border-width) solid var(--bs-navbar-toggler-border-color);border-radius:var(--bs-navbar-toggler-border-radius);transition:var(--bs-navbar-toggler-transition)}@media (prefers-reduced-motion:reduce){.navbar-toggler{transition:none}}.navbar-toggler:hover{text-decoration:none}.navbar-toggler:focus{text-decoration:none;outline:0;box-shadow:0 0 0 var(--bs-navbar-toggler-focus-width)}.navbar-toggler-icon{display:inline-block;width:1.5em;height:1.5em;vertical-align:middle;background-image:var(--bs-navbar-toggler-icon-bg);background-repeat:no-repeat;background-position:center;background-size:100%}.navbar-nav-scroll{max-height:var(--bs-scroll-height,75vh);overflow-y:auto}@media (min-width:576px){.navbar-expand-sm{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand-sm .navbar-nav{flex-direction:row}.navbar-expand-sm .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-sm .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand-sm .navbar-nav-scroll{overflow:visible}.navbar-expand-sm .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-sm .navbar-toggler{display:none}.navbar-expand-sm .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand-sm .offcanvas .offcanvas-header{display:none}.navbar-expand-sm .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}}@media (min-width:768px){.navbar-expand-md{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand-md .navbar-nav{flex-direction:row}.navbar-expand-md .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-md .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand-md .navbar-nav-scroll{overflow:visible}.navbar-expand-md .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-md .navbar-toggler{display:none}.navbar-expand-md .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand-md .offcanvas .offcanvas-header{display:none}.navbar-expand-md .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}}@media (min-width:992px){.navbar-expand-lg{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand-lg .navbar-nav{flex-direction:row}.navbar-expand-lg .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-lg .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand-lg .navbar-nav-scroll{overflow:visible}.navbar-expand-lg .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-lg .navbar-toggler{display:none}.navbar-expand-lg .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand-lg .offcanvas .offcanvas-header{display:none}.navbar-expand-lg .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}}@media (min-width:1200px){.navbar-expand-xl{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand-xl .navbar-nav{flex-direction:row}.navbar-expand-xl .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-xl .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand-xl .navbar-nav-scroll{overflow:visible}.navbar-expand-xl .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-xl .navbar-toggler{display:none}.navbar-expand-xl .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand-xl .offcanvas .offcanvas-header{display:none}.navbar-expand-xl .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}}@media (min-width:1400px){.navbar-expand-xxl{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand-xxl .navbar-nav{flex-direction:row}.navbar-expand-xxl .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-xxl .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand-xxl .navbar-nav-scroll{overflow:visible}.navbar-expand-xxl .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-xxl .navbar-toggler{display:none}.navbar-expand-xxl .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand-xxl .offcanvas .offcanvas-header{display:none}.navbar-expand-xxl .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}}.navbar-expand{flex-wrap:nowrap;justify-content:flex-start}.navbar-expand .navbar-nav{flex-direction:row}.navbar-expand .navbar-nav .dropdown-menu{position:absolute}.navbar-expand .navbar-nav .nav-link{padding-right:var(--bs-navbar-nav-link-padding-x);padding-left:var(--bs-navbar-nav-link-padding-x)}.navbar-expand .navbar-nav-scroll{overflow:visible}.navbar-expand .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand .navbar-toggler{display:none}.navbar-expand .offcanvas{position:static;z-index:auto;flex-grow:1;width:auto!important;height:auto!important;visibility:visible!important;background-color:transparent!important;border:0!important;transform:none!important;transition:none}.navbar-expand .offcanvas .offcanvas-header{display:none}.navbar-expand .offcanvas .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible}.navbar-dark,.navbar[data-bs-theme=dark]{--bs-navbar-color:rgba(255, 255, 255, 0.55);--bs-navbar-hover-color:rgba(255, 255, 255, 0.75);--bs-navbar-disabled-color:rgba(255, 255, 255, 0.25);--bs-navbar-active-color:#fff;--bs-navbar-brand-color:#fff;--bs-navbar-brand-hover-color:#fff;--bs-navbar-toggler-border-color:rgba(255, 255, 255, 0.1);--bs-navbar-toggler-icon-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 30 30'%3e%3cpath stroke='rgba%28255, 255, 255, 0.55%29' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/%3e%3c/svg%3e")}[data-bs-theme=dark] .navbar-toggler-icon{--bs-navbar-toggler-icon-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 30 30'%3e%3cpath stroke='rgba%28255, 255, 255, 0.55%29' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/%3e%3c/svg%3e")}.card{--bs-card-spacer-y:1rem;--bs-card-spacer-x:1rem;--bs-card-title-spacer-y:0.5rem;--bs-card-title-color: ;--bs-card-subtitle-color: ;--bs-card-border-width:var(--bs-border-width);--bs-card-border-color:var(--bs-border-color-translucent);--bs-card-border-radius:var(--bs-border-radius);--bs-card-box-shadow: ;--bs-card-inner-border-radius:calc(var(--bs-border-radius) - (var(--bs-border-width)));--bs-card-cap-padding-y:0.5rem;--bs-card-cap-padding-x:1rem;--bs-card-cap-bg:rgba(var(--bs-body-color-rgb), 0.03);--bs-card-cap-color: ;--bs-card-height: ;--bs-card-color: ;--bs-card-bg:var(--bs-body-bg);--bs-card-img-overlay-padding:1rem;--bs-card-group-margin:0.75rem;position:relative;display:flex;flex-direction:column;min-width:0;height:var(--bs-card-height);color:var(--bs-body-color);word-wrap:break-word;background-color:var(--bs-card-bg);background-clip:border-box;border:var(--bs-card-border-width) solid var(--bs-card-border-color);border-radius:var(--bs-card-border-radius)}.card>hr{margin-right:0;margin-left:0}.card>.list-group{border-top:inherit;border-bottom:inherit}.card>.list-group:first-child{border-top-width:0;border-top-left-radius:var(--bs-card-inner-border-radius);border-top-right-radius:var(--bs-card-inner-border-radius)}.card>.list-group:last-child{border-bottom-width:0;border-bottom-right-radius:var(--bs-card-inner-border-radius);border-bottom-left-radius:var(--bs-card-inner-border-radius)}.card>.card-header+.list-group,.card>.list-group+.card-footer{border-top:0}.card-body{flex:1 1 auto;padding:var(--bs-card-spacer-y) var(--bs-card-spacer-x);color:var(--bs-card-color)}.card-title{margin-bottom:var(--bs-card-title-spacer-y);color:var(--bs-card-title-color)}.card-subtitle{margin-top:calc(-.5 * var(--bs-card-title-spacer-y));margin-bottom:0;color:var(--bs-card-subtitle-color)}.card-text:last-child{margin-bottom:0}.card-link+.card-link{margin-left:var(--bs-card-spacer-x)}.card-header{padding:var(--bs-card-cap-padding-y) var(--bs-card-cap-padding-x);margin-bottom:0;color:var(--bs-card-cap-color);background-color:var(--bs-card-cap-bg);border-bottom:var(--bs-card-border-width) solid var(--bs-card-border-color)}.card-header:first-child{border-radius:var(--bs-card-inner-border-radius) var(--bs-card-inner-border-radius) 0 0}.card-footer{padding:var(--bs-card-cap-padding-y) var(--bs-card-cap-padding-x);color:var(--bs-card-cap-color);background-color:var(--bs-card-cap-bg);border-top:var(--bs-card-border-width) solid var(--bs-card-border-color)}.card-footer:last-child{border-radius:0 0 var(--bs-card-inner-border-radius) var(--bs-card-inner-border-radius)}.card-header-tabs{margin-right:calc(-.5 * var(--bs-card-cap-padding-x));margin-bottom:calc(-1 * var(--bs-card-cap-padding-y));margin-left:calc(-.5 * var(--bs-card-cap-padding-x));border-bottom:0}.card-header-tabs .nav-link.active{background-color:var(--bs-card-bg);border-bottom-color:var(--bs-card-bg)}.card-header-pills{margin-right:calc(-.5 * var(--bs-card-cap-padding-x));margin-left:calc(-.5 * var(--bs-card-cap-padding-x))}.card-img-overlay{position:absolute;top:0;right:0;bottom:0;left:0;padding:var(--bs-card-img-overlay-padding);border-radius:var(--bs-card-inner-border-radius)}.card-img,.card-img-bottom,.card-img-top{width:100%}.card-img,.card-img-top{border-top-left-radius:var(--bs-card-inner-border-radius);border-top-right-radius:var(--bs-card-inner-border-radius)}.card-img,.card-img-bottom{border-bottom-right-radius:var(--bs-card-inner-border-radius);border-bottom-left-radius:var(--bs-card-inner-border-radius)}.card-group>.card{margin-bottom:var(--bs-card-group-margin)}@media (min-width:576px){.card-group{display:flex;flex-flow:row wrap}.card-group>.card{flex:1 0 0%;margin-bottom:0}.card-group>.card+.card{margin-left:0;border-left:0}.card-group>.card:not(:last-child){border-top-right-radius:0;border-bottom-right-radius:0}.card-group>.card:not(:last-child) .card-header,.card-group>.card:not(:last-child) .card-img-top{border-top-right-radius:0}.card-group>.card:not(:last-child) .card-footer,.card-group>.card:not(:last-child) .card-img-bottom{border-bottom-right-radius:0}.card-group>.card:not(:first-child){border-top-left-radius:0;border-bottom-left-radius:0}.card-group>.card:not(:first-child) .card-header,.card-group>.card:not(:first-child) .card-img-top{border-top-left-radius:0}.card-group>.card:not(:first-child) .card-footer,.card-group>.card:not(:first-child) .card-img-bottom{border-bottom-left-radius:0}}.accordion{--bs-accordion-color:var(--bs-body-color);--bs-accordion-bg:var(--bs-body-bg);--bs-accordion-transition:color 0.15s ease-in-out,background-color 0.15s ease-in-out,border-color 0.15s ease-in-out,box-shadow 0.15s ease-in-out,border-radius 0.15s ease;--bs-accordion-border-color:var(--bs-border-color);--bs-accordion-border-width:var(--bs-border-width);--bs-accordion-border-radius:var(--bs-border-radius);--bs-accordion-inner-border-radius:calc(var(--bs-border-radius) - (var(--bs-border-width)));--bs-accordion-btn-padding-x:1.25rem;--bs-accordion-btn-padding-y:1rem;--bs-accordion-btn-color:var(--bs-body-color);--bs-accordion-btn-bg:var(--bs-accordion-bg);--bs-accordion-btn-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='none' stroke='%23212529' stroke-linecap='round' stroke-linejoin='round'%3e%3cpath d='M2 5L8 11L14 5'/%3e%3c/svg%3e");--bs-accordion-btn-icon-width:1.25rem;--bs-accordion-btn-icon-transform:rotate(-180deg);--bs-accordion-btn-icon-transition:transform 0.2s ease-in-out;--bs-accordion-btn-active-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='none' stroke='%23052c65' stroke-linecap='round' stroke-linejoin='round'%3e%3cpath d='M2 5L8 11L14 5'/%3e%3c/svg%3e");--bs-accordion-btn-focus-box-shadow:0 0 0 0.25rem rgba(13, 110, 253, 0.25);--bs-accordion-body-padding-x:1.25rem;--bs-accordion-body-padding-y:1rem;--bs-accordion-active-color:var(--bs-primary-text-emphasis);--bs-accordion-active-bg:var(--bs-primary-bg-subtle)}.accordion-button{position:relative;display:flex;align-items:center;width:100%;padding:var(--bs-accordion-btn-padding-y) var(--bs-accordion-btn-padding-x);font-size:1rem;color:var(--bs-accordion-btn-color);text-align:left;background-color:var(--bs-accordion-btn-bg);border:0;border-radius:0;overflow-anchor:none;transition:var(--bs-accordion-transition)}@media (prefers-reduced-motion:reduce){.accordion-button{transition:none}}.accordion-button:not(.collapsed){color:var(--bs-accordion-active-color);background-color:var(--bs-accordion-active-bg);box-shadow:inset 0 calc(-1 * var(--bs-accordion-border-width)) 0 var(--bs-accordion-border-color)}.accordion-button:not(.collapsed)::after{background-image:var(--bs-accordion-btn-active-icon);transform:var(--bs-accordion-btn-icon-transform)}.accordion-button::after{flex-shrink:0;width:var(--bs-accordion-btn-icon-width);height:var(--bs-accordion-btn-icon-width);margin-left:auto;content:"";background-image:var(--bs-accordion-btn-icon);background-repeat:no-repeat;background-size:var(--bs-accordion-btn-icon-width);transition:var(--bs-accordion-btn-icon-transition)}@media (prefers-reduced-motion:reduce){.accordion-button::after{transition:none}}.accordion-button:hover{z-index:2}.accordion-button:focus{z-index:3;outline:0;box-shadow:var(--bs-accordion-btn-focus-box-shadow)}.accordion-header{margin-bottom:0}.accordion-item{color:var(--bs-accordion-color);background-color:var(--bs-accordion-bg);border:var(--bs-accordion-border-width) solid var(--bs-accordion-border-color)}.accordion-item:first-of-type{border-top-left-radius:var(--bs-accordion-border-radius);border-top-right-radius:var(--bs-accordion-border-radius)}.accordion-item:first-of-type>.accordion-header .accordion-button{border-top-left-radius:var(--bs-accordion-inner-border-radius);border-top-right-radius:var(--bs-accordion-inner-border-radius)}.accordion-item:not(:first-of-type){border-top:0}.accordion-item:last-of-type{border-bottom-right-radius:var(--bs-accordion-border-radius);border-bottom-left-radius:var(--bs-accordion-border-radius)}.accordion-item:last-of-type>.accordion-header .accordion-button.collapsed{border-bottom-right-radius:var(--bs-accordion-inner-border-radius);border-bottom-left-radius:var(--bs-accordion-inner-border-radius)}.accordion-item:last-of-type>.accordion-collapse{border-bottom-right-radius:var(--bs-accordion-border-radius);border-bottom-left-radius:var(--bs-accordion-border-radius)}.accordion-body{padding:var(--bs-accordion-body-padding-y) var(--bs-accordion-body-padding-x)}.accordion-flush>.accordion-item{border-right:0;border-left:0;border-radius:0}.accordion-flush>.accordion-item:first-child{border-top:0}.accordion-flush>.accordion-item:last-child{border-bottom:0}.accordion-flush>.accordion-item>.accordion-header .accordion-button,.accordion-flush>.accordion-item>.accordion-header .accordion-button.collapsed{border-radius:0}.accordion-flush>.accordion-item>.accordion-collapse{border-radius:0}[data-bs-theme=dark] .accordion-button::after{--bs-accordion-btn-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%236ea8fe'%3e%3cpath fill-rule='evenodd' d='M1.646 4.646a.5.5 0 0 1 .708 0L8 10.293l5.646-5.647a.5.5 0 0 1 .708.708l-6 6a.5.5 0 0 1-.708 0l-6-6a.5.5 0 0 1 0-.708z'/%3e%3c/svg%3e");--bs-accordion-btn-active-icon:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%236ea8fe'%3e%3cpath fill-rule='evenodd' d='M1.646 4.646a.5.5 0 0 1 .708 0L8 10.293l5.646-5.647a.5.5 0 0 1 .708.708l-6 6a.5.5 0 0 1-.708 0l-6-6a.5.5 0 0 1 0-.708z'/%3e%3c/svg%3e")}.breadcrumb{--bs-breadcrumb-padding-x:0;--bs-breadcrumb-padding-y:0;--bs-breadcrumb-margin-bottom:1rem;--bs-breadcrumb-bg: ;--bs-breadcrumb-border-radius: ;--bs-breadcrumb-divider-color:var(--bs-secondary-color);--bs-breadcrumb-item-padding-x:0.5rem;--bs-breadcrumb-item-active-color:var(--bs-secondary-color);display:flex;flex-wrap:wrap;padding:var(--bs-breadcrumb-padding-y) var(--bs-breadcrumb-padding-x);margin-bottom:var(--bs-breadcrumb-margin-bottom);font-size:var(--bs-breadcrumb-font-size);list-style:none;background-color:var(--bs-breadcrumb-bg);border-radius:var(--bs-breadcrumb-border-radius)}.breadcrumb-item+.breadcrumb-item{padding-left:var(--bs-breadcrumb-item-padding-x)}.breadcrumb-item+.breadcrumb-item::before{float:left;padding-right:var(--bs-breadcrumb-item-padding-x);color:var(--bs-breadcrumb-divider-color);content:var(--bs-breadcrumb-divider, "/")}.breadcrumb-item.active{color:var(--bs-breadcrumb-item-active-color)}.pagination{--bs-pagination-padding-x:0.75rem;--bs-pagination-padding-y:0.375rem;--bs-pagination-font-size:1rem;--bs-pagination-color:var(--bs-link-color);--bs-pagination-bg:var(--bs-body-bg);--bs-pagination-border-width:var(--bs-border-width);--bs-pagination-border-color:var(--bs-border-color);--bs-pagination-border-radius:var(--bs-border-radius);--bs-pagination-hover-color:var(--bs-link-hover-color);--bs-pagination-hover-bg:var(--bs-tertiary-bg);--bs-pagination-hover-border-color:var(--bs-border-color);--bs-pagination-focus-color:var(--bs-link-hover-color);--bs-pagination-focus-bg:var(--bs-secondary-bg);--bs-pagination-focus-box-shadow:0 0 0 0.25rem rgba(13, 110, 253, 0.25);--bs-pagination-active-color:#fff;--bs-pagination-active-bg:#0d6efd;--bs-pagination-active-border-color:#0d6efd;--bs-pagination-disabled-color:var(--bs-secondary-color);--bs-pagination-disabled-bg:var(--bs-secondary-bg);--bs-pagination-disabled-border-color:var(--bs-border-color);display:flex;padding-left:0;list-style:none}.page-link{position:relative;display:block;padding:var(--bs-pagination-padding-y) var(--bs-pagination-padding-x);font-size:var(--bs-pagination-font-size);color:var(--bs-pagination-color);text-decoration:none;background-color:var(--bs-pagination-bg);border:var(--bs-pagination-border-width) solid var(--bs-pagination-border-color);transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.page-link{transition:none}}.page-link:hover{z-index:2;color:var(--bs-pagination-hover-color);background-color:var(--bs-pagination-hover-bg);border-color:var(--bs-pagination-hover-border-color)}.page-link:focus{z-index:3;color:var(--bs-pagination-focus-color);background-color:var(--bs-pagination-focus-bg);outline:0;box-shadow:var(--bs-pagination-focus-box-shadow)}.active>.page-link,.page-link.active{z-index:3;color:var(--bs-pagination-active-color);background-color:var(--bs-pagination-active-bg);border-color:var(--bs-pagination-active-border-color)}.disabled>.page-link,.page-link.disabled{color:var(--bs-pagination-disabled-color);pointer-events:none;background-color:var(--bs-pagination-disabled-bg);border-color:var(--bs-pagination-disabled-border-color)}.page-item:not(:first-child) .page-link{margin-left:calc(var(--bs-border-width) * -1)}.page-item:first-child .page-link{border-top-left-radius:var(--bs-pagination-border-radius);border-bottom-left-radius:var(--bs-pagination-border-radius)}.page-item:last-child .page-link{border-top-right-radius:var(--bs-pagination-border-radius);border-bottom-right-radius:var(--bs-pagination-border-radius)}.pagination-lg{--bs-pagination-padding-x:1.5rem;--bs-pagination-padding-y:0.75rem;--bs-pagination-font-size:1.25rem;--bs-pagination-border-radius:var(--bs-border-radius-lg)}.pagination-sm{--bs-pagination-padding-x:0.5rem;--bs-pagination-padding-y:0.25rem;--bs-pagination-font-size:0.875rem;--bs-pagination-border-radius:var(--bs-border-radius-sm)}.badge{--bs-badge-padding-x:0.65em;--bs-badge-padding-y:0.35em;--bs-badge-font-size:0.75em;--bs-badge-font-weight:700;--bs-badge-color:#fff;--bs-badge-border-radius:var(--bs-border-radius);display:inline-block;padding:var(--bs-badge-padding-y) var(--bs-badge-padding-x);font-size:var(--bs-badge-font-size);font-weight:var(--bs-badge-font-weight);line-height:1;color:var(--bs-badge-color);text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:var(--bs-badge-border-radius)}.badge:empty{display:none}.btn .badge{position:relative;top:-1px}.alert{--bs-alert-bg:transparent;--bs-alert-padding-x:1rem;--bs-alert-padding-y:1rem;--bs-alert-margin-bottom:1rem;--bs-alert-color:inherit;--bs-alert-border-color:transparent;--bs-alert-border:var(--bs-border-width) solid var(--bs-alert-border-color);--bs-alert-border-radius:var(--bs-border-radius);--bs-alert-link-color:inherit;position:relative;padding:var(--bs-alert-padding-y) var(--bs-alert-padding-x);margin-bottom:var(--bs-alert-margin-bottom);color:var(--bs-alert-color);background-color:var(--bs-alert-bg);border:var(--bs-alert-border);border-radius:var(--bs-alert-border-radius)}.alert-heading{color:inherit}.alert-link{font-weight:700;color:var(--bs-alert-link-color)}.alert-dismissible{padding-right:3rem}.alert-dismissible .btn-close{position:absolute;top:0;right:0;z-index:2;padding:1.25rem 1rem}.alert-primary{--bs-alert-color:var(--bs-primary-text-emphasis);--bs-alert-bg:var(--bs-primary-bg-subtle);--bs-alert-border-color:var(--bs-primary-border-subtle);--bs-alert-link-color:var(--bs-primary-text-emphasis)}.alert-secondary{--bs-alert-color:var(--bs-secondary-text-emphasis);--bs-alert-bg:var(--bs-secondary-bg-subtle);--bs-alert-border-color:var(--bs-secondary-border-subtle);--bs-alert-link-color:var(--bs-secondary-text-emphasis)}.alert-success{--bs-alert-color:var(--bs-success-text-emphasis);--bs-alert-bg:var(--bs-success-bg-subtle);--bs-alert-border-color:var(--bs-success-border-subtle);--bs-alert-link-color:var(--bs-success-text-emphasis)}.alert-info{--bs-alert-color:var(--bs-info-text-emphasis);--bs-alert-bg:var(--bs-info-bg-subtle);--bs-alert-border-color:var(--bs-info-border-subtle);--bs-alert-link-color:var(--bs-info-text-emphasis)}.alert-warning{--bs-alert-color:var(--bs-warning-text-emphasis);--bs-alert-bg:var(--bs-warning-bg-subtle);--bs-alert-border-color:var(--bs-warning-border-subtle);--bs-alert-link-color:var(--bs-warning-text-emphasis)}.alert-danger{--bs-alert-color:var(--bs-danger-text-emphasis);--bs-alert-bg:var(--bs-danger-bg-subtle);--bs-alert-border-color:var(--bs-danger-border-subtle);--bs-alert-link-color:var(--bs-danger-text-emphasis)}.alert-light{--bs-alert-color:var(--bs-light-text-emphasis);--bs-alert-bg:var(--bs-light-bg-subtle);--bs-alert-border-color:var(--bs-light-border-subtle);--bs-alert-link-color:var(--bs-light-text-emphasis)}.alert-dark{--bs-alert-color:var(--bs-dark-text-emphasis);--bs-alert-bg:var(--bs-dark-bg-subtle);--bs-alert-border-color:var(--bs-dark-border-subtle);--bs-alert-link-color:var(--bs-dark-text-emphasis)}@keyframes progress-bar-stripes{0%{background-position-x:1rem}}.progress,.progress-stacked{--bs-progress-height:1rem;--bs-progress-font-size:0.75rem;--bs-progress-bg:var(--bs-secondary-bg);--bs-progress-border-radius:var(--bs-border-radius);--bs-progress-box-shadow:var(--bs-box-shadow-inset);--bs-progress-bar-color:#fff;--bs-progress-bar-bg:#0d6efd;--bs-progress-bar-transition:width 0.6s ease;display:flex;height:var(--bs-progress-height);overflow:hidden;font-size:var(--bs-progress-font-size);background-color:var(--bs-progress-bg);border-radius:var(--bs-progress-border-radius)}.progress-bar{display:flex;flex-direction:column;justify-content:center;overflow:hidden;color:var(--bs-progress-bar-color);text-align:center;white-space:nowrap;background-color:var(--bs-progress-bar-bg);transition:var(--bs-progress-bar-transition)}@media (prefers-reduced-motion:reduce){.progress-bar{transition:none}}.progress-bar-striped{background-image:linear-gradient(45deg,rgba(255,255,255,.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,.15) 50%,rgba(255,255,255,.15) 75%,transparent 75%,transparent);background-size:var(--bs-progress-height) var(--bs-progress-height)}.progress-stacked>.progress{overflow:visible}.progress-stacked>.progress>.progress-bar{width:100%}.progress-bar-animated{animation:1s linear infinite progress-bar-stripes}@media (prefers-reduced-motion:reduce){.progress-bar-animated{animation:none}}.list-group{--bs-list-group-color:var(--bs-body-color);--bs-list-group-bg:var(--bs-body-bg);--bs-list-group-border-color:var(--bs-border-color);--bs-list-group-border-width:var(--bs-border-width);--bs-list-group-border-radius:var(--bs-border-radius);--bs-list-group-item-padding-x:1rem;--bs-list-group-item-padding-y:0.5rem;--bs-list-group-action-color:var(--bs-secondary-color);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-tertiary-bg);--bs-list-group-action-active-color:var(--bs-body-color);--bs-list-group-action-active-bg:var(--bs-secondary-bg);--bs-list-group-disabled-color:var(--bs-secondary-color);--bs-list-group-disabled-bg:var(--bs-body-bg);--bs-list-group-active-color:#fff;--bs-list-group-active-bg:#0d6efd;--bs-list-group-active-border-color:#0d6efd;display:flex;flex-direction:column;padding-left:0;margin-bottom:0;border-radius:var(--bs-list-group-border-radius)}.list-group-numbered{list-style-type:none;counter-reset:section}.list-group-numbered>.list-group-item::before{content:counters(section, ".") ". ";counter-increment:section}.list-group-item-action{width:100%;color:var(--bs-list-group-action-color);text-align:inherit}.list-group-item-action:focus,.list-group-item-action:hover{z-index:1;color:var(--bs-list-group-action-hover-color);text-decoration:none;background-color:var(--bs-list-group-action-hover-bg)}.list-group-item-action:active{color:var(--bs-list-group-action-active-color);background-color:var(--bs-list-group-action-active-bg)}.list-group-item{position:relative;display:block;padding:var(--bs-list-group-item-padding-y) var(--bs-list-group-item-padding-x);color:var(--bs-list-group-color);text-decoration:none;background-color:var(--bs-list-group-bg);border:var(--bs-list-group-border-width) solid var(--bs-list-group-border-color)}.list-group-item:first-child{border-top-left-radius:inherit;border-top-right-radius:inherit}.list-group-item:last-child{border-bottom-right-radius:inherit;border-bottom-left-radius:inherit}.list-group-item.disabled,.list-group-item:disabled{color:var(--bs-list-group-disabled-color);pointer-events:none;background-color:var(--bs-list-group-disabled-bg)}.list-group-item.active{z-index:2;color:var(--bs-list-group-active-color);background-color:var(--bs-list-group-active-bg);border-color:var(--bs-list-group-active-border-color)}.list-group-item+.list-group-item{border-top-width:0}.list-group-item+.list-group-item.active{margin-top:calc(-1 * var(--bs-list-group-border-width));border-top-width:var(--bs-list-group-border-width)}.list-group-horizontal{flex-direction:row}.list-group-horizontal>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal>.list-group-item.active{margin-top:0}.list-group-horizontal>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}@media (min-width:576px){.list-group-horizontal-sm{flex-direction:row}.list-group-horizontal-sm>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal-sm>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal-sm>.list-group-item.active{margin-top:0}.list-group-horizontal-sm>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal-sm>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}}@media (min-width:768px){.list-group-horizontal-md{flex-direction:row}.list-group-horizontal-md>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal-md>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal-md>.list-group-item.active{margin-top:0}.list-group-horizontal-md>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal-md>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}}@media (min-width:992px){.list-group-horizontal-lg{flex-direction:row}.list-group-horizontal-lg>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal-lg>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal-lg>.list-group-item.active{margin-top:0}.list-group-horizontal-lg>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal-lg>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}}@media (min-width:1200px){.list-group-horizontal-xl{flex-direction:row}.list-group-horizontal-xl>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal-xl>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal-xl>.list-group-item.active{margin-top:0}.list-group-horizontal-xl>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal-xl>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}}@media (min-width:1400px){.list-group-horizontal-xxl{flex-direction:row}.list-group-horizontal-xxl>.list-group-item:first-child:not(:last-child){border-bottom-left-radius:var(--bs-list-group-border-radius);border-top-right-radius:0}.list-group-horizontal-xxl>.list-group-item:last-child:not(:first-child){border-top-right-radius:var(--bs-list-group-border-radius);border-bottom-left-radius:0}.list-group-horizontal-xxl>.list-group-item.active{margin-top:0}.list-group-horizontal-xxl>.list-group-item+.list-group-item{border-top-width:var(--bs-list-group-border-width);border-left-width:0}.list-group-horizontal-xxl>.list-group-item+.list-group-item.active{margin-left:calc(-1 * var(--bs-list-group-border-width));border-left-width:var(--bs-list-group-border-width)}}.list-group-flush{border-radius:0}.list-group-flush>.list-group-item{border-width:0 0 var(--bs-list-group-border-width)}.list-group-flush>.list-group-item:last-child{border-bottom-width:0}.list-group-item-primary{--bs-list-group-color:var(--bs-primary-text-emphasis);--bs-list-group-bg:var(--bs-primary-bg-subtle);--bs-list-group-border-color:var(--bs-primary-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-primary-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-primary-border-subtle);--bs-list-group-active-color:var(--bs-primary-bg-subtle);--bs-list-group-active-bg:var(--bs-primary-text-emphasis);--bs-list-group-active-border-color:var(--bs-primary-text-emphasis)}.list-group-item-secondary{--bs-list-group-color:var(--bs-secondary-text-emphasis);--bs-list-group-bg:var(--bs-secondary-bg-subtle);--bs-list-group-border-color:var(--bs-secondary-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-secondary-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-secondary-border-subtle);--bs-list-group-active-color:var(--bs-secondary-bg-subtle);--bs-list-group-active-bg:var(--bs-secondary-text-emphasis);--bs-list-group-active-border-color:var(--bs-secondary-text-emphasis)}.list-group-item-success{--bs-list-group-color:var(--bs-success-text-emphasis);--bs-list-group-bg:var(--bs-success-bg-subtle);--bs-list-group-border-color:var(--bs-success-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-success-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-success-border-subtle);--bs-list-group-active-color:var(--bs-success-bg-subtle);--bs-list-group-active-bg:var(--bs-success-text-emphasis);--bs-list-group-active-border-color:var(--bs-success-text-emphasis)}.list-group-item-info{--bs-list-group-color:var(--bs-info-text-emphasis);--bs-list-group-bg:var(--bs-info-bg-subtle);--bs-list-group-border-color:var(--bs-info-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-info-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-info-border-subtle);--bs-list-group-active-color:var(--bs-info-bg-subtle);--bs-list-group-active-bg:var(--bs-info-text-emphasis);--bs-list-group-active-border-color:var(--bs-info-text-emphasis)}.list-group-item-warning{--bs-list-group-color:var(--bs-warning-text-emphasis);--bs-list-group-bg:var(--bs-warning-bg-subtle);--bs-list-group-border-color:var(--bs-warning-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-warning-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-warning-border-subtle);--bs-list-group-active-color:var(--bs-warning-bg-subtle);--bs-list-group-active-bg:var(--bs-warning-text-emphasis);--bs-list-group-active-border-color:var(--bs-warning-text-emphasis)}.list-group-item-danger{--bs-list-group-color:var(--bs-danger-text-emphasis);--bs-list-group-bg:var(--bs-danger-bg-subtle);--bs-list-group-border-color:var(--bs-danger-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-danger-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-danger-border-subtle);--bs-list-group-active-color:var(--bs-danger-bg-subtle);--bs-list-group-active-bg:var(--bs-danger-text-emphasis);--bs-list-group-active-border-color:var(--bs-danger-text-emphasis)}.list-group-item-light{--bs-list-group-color:var(--bs-light-text-emphasis);--bs-list-group-bg:var(--bs-light-bg-subtle);--bs-list-group-border-color:var(--bs-light-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-light-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-light-border-subtle);--bs-list-group-active-color:var(--bs-light-bg-subtle);--bs-list-group-active-bg:var(--bs-light-text-emphasis);--bs-list-group-active-border-color:var(--bs-light-text-emphasis)}.list-group-item-dark{--bs-list-group-color:var(--bs-dark-text-emphasis);--bs-list-group-bg:var(--bs-dark-bg-subtle);--bs-list-group-border-color:var(--bs-dark-border-subtle);--bs-list-group-action-hover-color:var(--bs-emphasis-color);--bs-list-group-action-hover-bg:var(--bs-dark-border-subtle);--bs-list-group-action-active-color:var(--bs-emphasis-color);--bs-list-group-action-active-bg:var(--bs-dark-border-subtle);--bs-list-group-active-color:var(--bs-dark-bg-subtle);--bs-list-group-active-bg:var(--bs-dark-text-emphasis);--bs-list-group-active-border-color:var(--bs-dark-text-emphasis)}.btn-close{--bs-btn-close-color:#000;--bs-btn-close-bg:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23000'%3e%3cpath d='M.293.293a1 1 0 0 1 1.414 0L8 6.586 14.293.293a1 1 0 1 1 1.414 1.414L9.414 8l6.293 6.293a1 1 0 0 1-1.414 1.414L8 9.414l-6.293 6.293a1 1 0 0 1-1.414-1.414L6.586 8 .293 1.707a1 1 0 0 1 0-1.414z'/%3e%3c/svg%3e");--bs-btn-close-opacity:0.5;--bs-btn-close-hover-opacity:0.75;--bs-btn-close-focus-shadow:0 0 0 0.25rem rgba(13, 110, 253, 0.25);--bs-btn-close-focus-opacity:1;--bs-btn-close-disabled-opacity:0.25;--bs-btn-close-white-filter:invert(1) grayscale(100%) brightness(200%);box-sizing:content-box;width:1em;height:1em;padding:.25em .25em;color:var(--bs-btn-close-color);background:transparent var(--bs-btn-close-bg) center/1em auto no-repeat;border:0;border-radius:.375rem;opacity:var(--bs-btn-close-opacity)}.btn-close:hover{color:var(--bs-btn-close-color);text-decoration:none;opacity:var(--bs-btn-close-hover-opacity)}.btn-close:focus{outline:0;box-shadow:var(--bs-btn-close-focus-shadow);opacity:var(--bs-btn-close-focus-opacity)}.btn-close.disabled,.btn-close:disabled{pointer-events:none;-webkit-user-select:none;-moz-user-select:none;user-select:none;opacity:var(--bs-btn-close-disabled-opacity)}.btn-close-white{filter:var(--bs-btn-close-white-filter)}[data-bs-theme=dark] .btn-close{filter:var(--bs-btn-close-white-filter)}.toast{--bs-toast-zindex:1090;--bs-toast-padding-x:0.75rem;--bs-toast-padding-y:0.5rem;--bs-toast-spacing:1.5rem;--bs-toast-max-width:350px;--bs-toast-font-size:0.875rem;--bs-toast-color: ;--bs-toast-bg:rgba(var(--bs-body-bg-rgb), 0.85);--bs-toast-border-width:var(--bs-border-width);--bs-toast-border-color:var(--bs-border-color-translucent);--bs-toast-border-radius:var(--bs-border-radius);--bs-toast-box-shadow:var(--bs-box-shadow);--bs-toast-header-color:var(--bs-secondary-color);--bs-toast-header-bg:rgba(var(--bs-body-bg-rgb), 0.85);--bs-toast-header-border-color:var(--bs-border-color-translucent);width:var(--bs-toast-max-width);max-width:100%;font-size:var(--bs-toast-font-size);color:var(--bs-toast-color);pointer-events:auto;background-color:var(--bs-toast-bg);background-clip:padding-box;border:var(--bs-toast-border-width) solid var(--bs-toast-border-color);box-shadow:var(--bs-toast-box-shadow);border-radius:var(--bs-toast-border-radius)}.toast.showing{opacity:0}.toast:not(.show){display:none}.toast-container{--bs-toast-zindex:1090;position:absolute;z-index:var(--bs-toast-zindex);width:-webkit-max-content;width:-moz-max-content;width:max-content;max-width:100%;pointer-events:none}.toast-container>:not(:last-child){margin-bottom:var(--bs-toast-spacing)}.toast-header{display:flex;align-items:center;padding:var(--bs-toast-padding-y) var(--bs-toast-padding-x);color:var(--bs-toast-header-color);background-color:var(--bs-toast-header-bg);background-clip:padding-box;border-bottom:var(--bs-toast-border-width) solid var(--bs-toast-header-border-color);border-top-left-radius:calc(var(--bs-toast-border-radius) - var(--bs-toast-border-width));border-top-right-radius:calc(var(--bs-toast-border-radius) - var(--bs-toast-border-width))}.toast-header .btn-close{margin-right:calc(-.5 * var(--bs-toast-padding-x));margin-left:var(--bs-toast-padding-x)}.toast-body{padding:var(--bs-toast-padding-x);word-wrap:break-word}.modal{--bs-modal-zindex:1055;--bs-modal-width:500px;--bs-modal-padding:1rem;--bs-modal-margin:0.5rem;--bs-modal-color: ;--bs-modal-bg:var(--bs-body-bg);--bs-modal-border-color:var(--bs-border-color-translucent);--bs-modal-border-width:var(--bs-border-width);--bs-modal-border-radius:var(--bs-border-radius-lg);--bs-modal-box-shadow:var(--bs-box-shadow-sm);--bs-modal-inner-border-radius:calc(var(--bs-border-radius-lg) - (var(--bs-border-width)));--bs-modal-header-padding-x:1rem;--bs-modal-header-padding-y:1rem;--bs-modal-header-padding:1rem 1rem;--bs-modal-header-border-color:var(--bs-border-color);--bs-modal-header-border-width:var(--bs-border-width);--bs-modal-title-line-height:1.5;--bs-modal-footer-gap:0.5rem;--bs-modal-footer-bg: ;--bs-modal-footer-border-color:var(--bs-border-color);--bs-modal-footer-border-width:var(--bs-border-width);position:fixed;top:0;left:0;z-index:var(--bs-modal-zindex);display:none;width:100%;height:100%;overflow-x:hidden;overflow-y:auto;outline:0}.modal-dialog{position:relative;width:auto;margin:var(--bs-modal-margin);pointer-events:none}.modal.fade .modal-dialog{transition:transform .3s ease-out;transform:translate(0,-50px)}@media (prefers-reduced-motion:reduce){.modal.fade .modal-dialog{transition:none}}.modal.show .modal-dialog{transform:none}.modal.modal-static .modal-dialog{transform:scale(1.02)}.modal-dialog-scrollable{height:calc(100% - var(--bs-modal-margin) * 2)}.modal-dialog-scrollable .modal-content{max-height:100%;overflow:hidden}.modal-dialog-scrollable .modal-body{overflow-y:auto}.modal-dialog-centered{display:flex;align-items:center;min-height:calc(100% - var(--bs-modal-margin) * 2)}.modal-content{position:relative;display:flex;flex-direction:column;width:100%;color:var(--bs-modal-color);pointer-events:auto;background-color:var(--bs-modal-bg);background-clip:padding-box;border:var(--bs-modal-border-width) solid var(--bs-modal-border-color);border-radius:var(--bs-modal-border-radius);outline:0}.modal-backdrop{--bs-backdrop-zindex:1050;--bs-backdrop-bg:#000;--bs-backdrop-opacity:0.5;position:fixed;top:0;left:0;z-index:var(--bs-backdrop-zindex);width:100vw;height:100vh;background-color:var(--bs-backdrop-bg)}.modal-backdrop.fade{opacity:0}.modal-backdrop.show{opacity:var(--bs-backdrop-opacity)}.modal-header{display:flex;flex-shrink:0;align-items:center;padding:var(--bs-modal-header-padding);border-bottom:var(--bs-modal-header-border-width) solid var(--bs-modal-header-border-color);border-top-left-radius:var(--bs-modal-inner-border-radius);border-top-right-radius:var(--bs-modal-inner-border-radius)}.modal-header .btn-close{padding:calc(var(--bs-modal-header-padding-y) * .5) calc(var(--bs-modal-header-padding-x) * .5);margin:calc(-.5 * var(--bs-modal-header-padding-y)) calc(-.5 * var(--bs-modal-header-padding-x)) calc(-.5 * var(--bs-modal-header-padding-y)) auto}.modal-title{margin-bottom:0;line-height:var(--bs-modal-title-line-height)}.modal-body{position:relative;flex:1 1 auto;padding:var(--bs-modal-padding)}.modal-footer{display:flex;flex-shrink:0;flex-wrap:wrap;align-items:center;justify-content:flex-end;padding:calc(var(--bs-modal-padding) - var(--bs-modal-footer-gap) * .5);background-color:var(--bs-modal-footer-bg);border-top:var(--bs-modal-footer-border-width) solid var(--bs-modal-footer-border-color);border-bottom-right-radius:var(--bs-modal-inner-border-radius);border-bottom-left-radius:var(--bs-modal-inner-border-radius)}.modal-footer>*{margin:calc(var(--bs-modal-footer-gap) * .5)}@media (min-width:576px){.modal{--bs-modal-margin:1.75rem;--bs-modal-box-shadow:var(--bs-box-shadow)}.modal-dialog{max-width:var(--bs-modal-width);margin-right:auto;margin-left:auto}.modal-sm{--bs-modal-width:300px}}@media (min-width:992px){.modal-lg,.modal-xl{--bs-modal-width:800px}}@media (min-width:1200px){.modal-xl{--bs-modal-width:1140px}}.modal-fullscreen{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen .modal-footer,.modal-fullscreen .modal-header{border-radius:0}.modal-fullscreen .modal-body{overflow-y:auto}@media (max-width:575.98px){.modal-fullscreen-sm-down{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen-sm-down .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen-sm-down .modal-footer,.modal-fullscreen-sm-down .modal-header{border-radius:0}.modal-fullscreen-sm-down .modal-body{overflow-y:auto}}@media (max-width:767.98px){.modal-fullscreen-md-down{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen-md-down .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen-md-down .modal-footer,.modal-fullscreen-md-down .modal-header{border-radius:0}.modal-fullscreen-md-down .modal-body{overflow-y:auto}}@media (max-width:991.98px){.modal-fullscreen-lg-down{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen-lg-down .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen-lg-down .modal-footer,.modal-fullscreen-lg-down .modal-header{border-radius:0}.modal-fullscreen-lg-down .modal-body{overflow-y:auto}}@media (max-width:1199.98px){.modal-fullscreen-xl-down{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen-xl-down .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen-xl-down .modal-footer,.modal-fullscreen-xl-down .modal-header{border-radius:0}.modal-fullscreen-xl-down .modal-body{overflow-y:auto}}@media (max-width:1399.98px){.modal-fullscreen-xxl-down{width:100vw;max-width:none;height:100%;margin:0}.modal-fullscreen-xxl-down .modal-content{height:100%;border:0;border-radius:0}.modal-fullscreen-xxl-down .modal-footer,.modal-fullscreen-xxl-down .modal-header{border-radius:0}.modal-fullscreen-xxl-down .modal-body{overflow-y:auto}}.tooltip{--bs-tooltip-zindex:1080;--bs-tooltip-max-width:200px;--bs-tooltip-padding-x:0.5rem;--bs-tooltip-padding-y:0.25rem;--bs-tooltip-margin: ;--bs-tooltip-font-size:0.875rem;--bs-tooltip-color:var(--bs-body-bg);--bs-tooltip-bg:var(--bs-emphasis-color);--bs-tooltip-border-radius:var(--bs-border-radius);--bs-tooltip-opacity:0.9;--bs-tooltip-arrow-width:0.8rem;--bs-tooltip-arrow-height:0.4rem;z-index:var(--bs-tooltip-zindex);display:block;margin:var(--bs-tooltip-margin);font-family:var(--bs-font-sans-serif);font-style:normal;font-weight:400;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;letter-spacing:normal;word-break:normal;white-space:normal;word-spacing:normal;line-break:auto;font-size:var(--bs-tooltip-font-size);word-wrap:break-word;opacity:0}.tooltip.show{opacity:var(--bs-tooltip-opacity)}.tooltip .tooltip-arrow{display:block;width:var(--bs-tooltip-arrow-width);height:var(--bs-tooltip-arrow-height)}.tooltip .tooltip-arrow::before{position:absolute;content:"";border-color:transparent;border-style:solid}.bs-tooltip-auto[data-popper-placement^=top] .tooltip-arrow,.bs-tooltip-top .tooltip-arrow{bottom:calc(-1 * var(--bs-tooltip-arrow-height))}.bs-tooltip-auto[data-popper-placement^=top] .tooltip-arrow::before,.bs-tooltip-top .tooltip-arrow::before{top:-1px;border-width:var(--bs-tooltip-arrow-height) calc(var(--bs-tooltip-arrow-width) * .5) 0;border-top-color:var(--bs-tooltip-bg)}.bs-tooltip-auto[data-popper-placement^=right] .tooltip-arrow,.bs-tooltip-end .tooltip-arrow{left:calc(-1 * var(--bs-tooltip-arrow-height));width:var(--bs-tooltip-arrow-height);height:var(--bs-tooltip-arrow-width)}.bs-tooltip-auto[data-popper-placement^=right] .tooltip-arrow::before,.bs-tooltip-end .tooltip-arrow::before{right:-1px;border-width:calc(var(--bs-tooltip-arrow-width) * .5) var(--bs-tooltip-arrow-height) calc(var(--bs-tooltip-arrow-width) * .5) 0;border-right-color:var(--bs-tooltip-bg)}.bs-tooltip-auto[data-popper-placement^=bottom] .tooltip-arrow,.bs-tooltip-bottom .tooltip-arrow{top:calc(-1 * var(--bs-tooltip-arrow-height))}.bs-tooltip-auto[data-popper-placement^=bottom] .tooltip-arrow::before,.bs-tooltip-bottom .tooltip-arrow::before{bottom:-1px;border-width:0 calc(var(--bs-tooltip-arrow-width) * .5) var(--bs-tooltip-arrow-height);border-bottom-color:var(--bs-tooltip-bg)}.bs-tooltip-auto[data-popper-placement^=left] .tooltip-arrow,.bs-tooltip-start .tooltip-arrow{right:calc(-1 * var(--bs-tooltip-arrow-height));width:var(--bs-tooltip-arrow-height);height:var(--bs-tooltip-arrow-width)}.bs-tooltip-auto[data-popper-placement^=left] .tooltip-arrow::before,.bs-tooltip-start .tooltip-arrow::before{left:-1px;border-width:calc(var(--bs-tooltip-arrow-width) * .5) 0 calc(var(--bs-tooltip-arrow-width) * .5) var(--bs-tooltip-arrow-height);border-left-color:var(--bs-tooltip-bg)}.tooltip-inner{max-width:var(--bs-tooltip-max-width);padding:var(--bs-tooltip-padding-y) var(--bs-tooltip-padding-x);color:var(--bs-tooltip-color);text-align:center;background-color:var(--bs-tooltip-bg);border-radius:var(--bs-tooltip-border-radius)}.popover{--bs-popover-zindex:1070;--bs-popover-max-width:276px;--bs-popover-font-size:0.875rem;--bs-popover-bg:var(--bs-body-bg);--bs-popover-border-width:var(--bs-border-width);--bs-popover-border-color:var(--bs-border-color-translucent);--bs-popover-border-radius:var(--bs-border-radius-lg);--bs-popover-inner-border-radius:calc(var(--bs-border-radius-lg) - var(--bs-border-width));--bs-popover-box-shadow:var(--bs-box-shadow);--bs-popover-header-padding-x:1rem;--bs-popover-header-padding-y:0.5rem;--bs-popover-header-font-size:1rem;--bs-popover-header-color:inherit;--bs-popover-header-bg:var(--bs-secondary-bg);--bs-popover-body-padding-x:1rem;--bs-popover-body-padding-y:1rem;--bs-popover-body-color:var(--bs-body-color);--bs-popover-arrow-width:1rem;--bs-popover-arrow-height:0.5rem;--bs-popover-arrow-border:var(--bs-popover-border-color);z-index:var(--bs-popover-zindex);display:block;max-width:var(--bs-popover-max-width);font-family:var(--bs-font-sans-serif);font-style:normal;font-weight:400;line-height:1.5;text-align:left;text-align:start;text-decoration:none;text-shadow:none;text-transform:none;letter-spacing:normal;word-break:normal;white-space:normal;word-spacing:normal;line-break:auto;font-size:var(--bs-popover-font-size);word-wrap:break-word;background-color:var(--bs-popover-bg);background-clip:padding-box;border:var(--bs-popover-border-width) solid var(--bs-popover-border-color);border-radius:var(--bs-popover-border-radius)}.popover .popover-arrow{display:block;width:var(--bs-popover-arrow-width);height:var(--bs-popover-arrow-height)}.popover .popover-arrow::after,.popover .popover-arrow::before{position:absolute;display:block;content:"";border-color:transparent;border-style:solid;border-width:0}.bs-popover-auto[data-popper-placement^=top]>.popover-arrow,.bs-popover-top>.popover-arrow{bottom:calc(-1 * (var(--bs-popover-arrow-height)) - var(--bs-popover-border-width))}.bs-popover-auto[data-popper-placement^=top]>.popover-arrow::after,.bs-popover-auto[data-popper-placement^=top]>.popover-arrow::before,.bs-popover-top>.popover-arrow::after,.bs-popover-top>.popover-arrow::before{border-width:var(--bs-popover-arrow-height) calc(var(--bs-popover-arrow-width) * .5) 0}.bs-popover-auto[data-popper-placement^=top]>.popover-arrow::before,.bs-popover-top>.popover-arrow::before{bottom:0;border-top-color:var(--bs-popover-arrow-border)}.bs-popover-auto[data-popper-placement^=top]>.popover-arrow::after,.bs-popover-top>.popover-arrow::after{bottom:var(--bs-popover-border-width);border-top-color:var(--bs-popover-bg)}.bs-popover-auto[data-popper-placement^=right]>.popover-arrow,.bs-popover-end>.popover-arrow{left:calc(-1 * (var(--bs-popover-arrow-height)) - var(--bs-popover-border-width));width:var(--bs-popover-arrow-height);height:var(--bs-popover-arrow-width)}.bs-popover-auto[data-popper-placement^=right]>.popover-arrow::after,.bs-popover-auto[data-popper-placement^=right]>.popover-arrow::before,.bs-popover-end>.popover-arrow::after,.bs-popover-end>.popover-arrow::before{border-width:calc(var(--bs-popover-arrow-width) * .5) var(--bs-popover-arrow-height) calc(var(--bs-popover-arrow-width) * .5) 0}.bs-popover-auto[data-popper-placement^=right]>.popover-arrow::before,.bs-popover-end>.popover-arrow::before{left:0;border-right-color:var(--bs-popover-arrow-border)}.bs-popover-auto[data-popper-placement^=right]>.popover-arrow::after,.bs-popover-end>.popover-arrow::after{left:var(--bs-popover-border-width);border-right-color:var(--bs-popover-bg)}.bs-popover-auto[data-popper-placement^=bottom]>.popover-arrow,.bs-popover-bottom>.popover-arrow{top:calc(-1 * (var(--bs-popover-arrow-height)) - var(--bs-popover-border-width))}.bs-popover-auto[data-popper-placement^=bottom]>.popover-arrow::after,.bs-popover-auto[data-popper-placement^=bottom]>.popover-arrow::before,.bs-popover-bottom>.popover-arrow::after,.bs-popover-bottom>.popover-arrow::before{border-width:0 calc(var(--bs-popover-arrow-width) * .5) var(--bs-popover-arrow-height)}.bs-popover-auto[data-popper-placement^=bottom]>.popover-arrow::before,.bs-popover-bottom>.popover-arrow::before{top:0;border-bottom-color:var(--bs-popover-arrow-border)}.bs-popover-auto[data-popper-placement^=bottom]>.popover-arrow::after,.bs-popover-bottom>.popover-arrow::after{top:var(--bs-popover-border-width);border-bottom-color:var(--bs-popover-bg)}.bs-popover-auto[data-popper-placement^=bottom] .popover-header::before,.bs-popover-bottom .popover-header::before{position:absolute;top:0;left:50%;display:block;width:var(--bs-popover-arrow-width);margin-left:calc(-.5 * var(--bs-popover-arrow-width));content:"";border-bottom:var(--bs-popover-border-width) solid var(--bs-popover-header-bg)}.bs-popover-auto[data-popper-placement^=left]>.popover-arrow,.bs-popover-start>.popover-arrow{right:calc(-1 * (var(--bs-popover-arrow-height)) - var(--bs-popover-border-width));width:var(--bs-popover-arrow-height);height:var(--bs-popover-arrow-width)}.bs-popover-auto[data-popper-placement^=left]>.popover-arrow::after,.bs-popover-auto[data-popper-placement^=left]>.popover-arrow::before,.bs-popover-start>.popover-arrow::after,.bs-popover-start>.popover-arrow::before{border-width:calc(var(--bs-popover-arrow-width) * .5) 0 calc(var(--bs-popover-arrow-width) * .5) var(--bs-popover-arrow-height)}.bs-popover-auto[data-popper-placement^=left]>.popover-arrow::before,.bs-popover-start>.popover-arrow::before{right:0;border-left-color:var(--bs-popover-arrow-border)}.bs-popover-auto[data-popper-placement^=left]>.popover-arrow::after,.bs-popover-start>.popover-arrow::after{right:var(--bs-popover-border-width);border-left-color:var(--bs-popover-bg)}.popover-header{padding:var(--bs-popover-header-padding-y) var(--bs-popover-header-padding-x);margin-bottom:0;font-size:var(--bs-popover-header-font-size);color:var(--bs-popover-header-color);background-color:var(--bs-popover-header-bg);border-bottom:var(--bs-popover-border-width) solid var(--bs-popover-border-color);border-top-left-radius:var(--bs-popover-inner-border-radius);border-top-right-radius:var(--bs-popover-inner-border-radius)}.popover-header:empty{display:none}.popover-body{padding:var(--bs-popover-body-padding-y) var(--bs-popover-body-padding-x);color:var(--bs-popover-body-color)}.carousel{position:relative}.carousel.pointer-event{touch-action:pan-y}.carousel-inner{position:relative;width:100%;overflow:hidden}.carousel-inner::after{display:block;clear:both;content:""}.carousel-item{position:relative;display:none;float:left;width:100%;margin-right:-100%;-webkit-backface-visibility:hidden;backface-visibility:hidden;transition:transform .6s ease-in-out}@media (prefers-reduced-motion:reduce){.carousel-item{transition:none}}.carousel-item-next,.carousel-item-prev,.carousel-item.active{display:block}.active.carousel-item-end,.carousel-item-next:not(.carousel-item-start){transform:translateX(100%)}.active.carousel-item-start,.carousel-item-prev:not(.carousel-item-end){transform:translateX(-100%)}.carousel-fade .carousel-item{opacity:0;transition-property:opacity;transform:none}.carousel-fade .carousel-item-next.carousel-item-start,.carousel-fade .carousel-item-prev.carousel-item-end,.carousel-fade .carousel-item.active{z-index:1;opacity:1}.carousel-fade .active.carousel-item-end,.carousel-fade .active.carousel-item-start{z-index:0;opacity:0;transition:opacity 0s .6s}@media (prefers-reduced-motion:reduce){.carousel-fade .active.carousel-item-end,.carousel-fade .active.carousel-item-start{transition:none}}.carousel-control-next,.carousel-control-prev{position:absolute;top:0;bottom:0;z-index:1;display:flex;align-items:center;justify-content:center;width:15%;padding:0;color:#fff;text-align:center;background:0 0;border:0;opacity:.5;transition:opacity .15s ease}@media (prefers-reduced-motion:reduce){.carousel-control-next,.carousel-control-prev{transition:none}}.carousel-control-next:focus,.carousel-control-next:hover,.carousel-control-prev:focus,.carousel-control-prev:hover{color:#fff;text-decoration:none;outline:0;opacity:.9}.carousel-control-prev{left:0}.carousel-control-next{right:0}.carousel-control-next-icon,.carousel-control-prev-icon{display:inline-block;width:2rem;height:2rem;background-repeat:no-repeat;background-position:50%;background-size:100% 100%}.carousel-control-prev-icon{background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3e%3cpath d='M11.354 1.646a.5.5 0 0 1 0 .708L5.707 8l5.647 5.646a.5.5 0 0 1-.708.708l-6-6a.5.5 0 0 1 0-.708l6-6a.5.5 0 0 1 .708 0z'/%3e%3c/svg%3e")}.carousel-control-next-icon{background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3e%3cpath d='M4.646 1.646a.5.5 0 0 1 .708 0l6 6a.5.5 0 0 1 0 .708l-6 6a.5.5 0 0 1-.708-.708L10.293 8 4.646 2.354a.5.5 0 0 1 0-.708z'/%3e%3c/svg%3e")}.carousel-indicators{position:absolute;right:0;bottom:0;left:0;z-index:2;display:flex;justify-content:center;padding:0;margin-right:15%;margin-bottom:1rem;margin-left:15%}.carousel-indicators [data-bs-target]{box-sizing:content-box;flex:0 1 auto;width:30px;height:3px;padding:0;margin-right:3px;margin-left:3px;text-indent:-999px;cursor:pointer;background-color:#fff;background-clip:padding-box;border:0;border-top:10px solid transparent;border-bottom:10px solid transparent;opacity:.5;transition:opacity .6s ease}@media (prefers-reduced-motion:reduce){.carousel-indicators [data-bs-target]{transition:none}}.carousel-indicators .active{opacity:1}.carousel-caption{position:absolute;right:15%;bottom:1.25rem;left:15%;padding-top:1.25rem;padding-bottom:1.25rem;color:#fff;text-align:center}.carousel-dark .carousel-control-next-icon,.carousel-dark .carousel-control-prev-icon{filter:invert(1) grayscale(100)}.carousel-dark .carousel-indicators [data-bs-target]{background-color:#000}.carousel-dark .carousel-caption{color:#000}[data-bs-theme=dark] .carousel .carousel-control-next-icon,[data-bs-theme=dark] .carousel .carousel-control-prev-icon,[data-bs-theme=dark].carousel .carousel-control-next-icon,[data-bs-theme=dark].carousel .carousel-control-prev-icon{filter:invert(1) grayscale(100)}[data-bs-theme=dark] .carousel .carousel-indicators [data-bs-target],[data-bs-theme=dark].carousel .carousel-indicators [data-bs-target]{background-color:#000}[data-bs-theme=dark] .carousel .carousel-caption,[data-bs-theme=dark].carousel .carousel-caption{color:#000}.spinner-border,.spinner-grow{display:inline-block;width:var(--bs-spinner-width);height:var(--bs-spinner-height);vertical-align:var(--bs-spinner-vertical-align);border-radius:50%;animation:var(--bs-spinner-animation-speed) linear infinite var(--bs-spinner-animation-name)}@keyframes spinner-border{to{transform:rotate(360deg)}}.spinner-border{--bs-spinner-width:2rem;--bs-spinner-height:2rem;--bs-spinner-vertical-align:-0.125em;--bs-spinner-border-width:0.25em;--bs-spinner-animation-speed:0.75s;--bs-spinner-animation-name:spinner-border;border:var(--bs-spinner-border-width) solid currentcolor;border-right-color:transparent}.spinner-border-sm{--bs-spinner-width:1rem;--bs-spinner-height:1rem;--bs-spinner-border-width:0.2em}@keyframes spinner-grow{0%{transform:scale(0)}50%{opacity:1;transform:none}}.spinner-grow{--bs-spinner-width:2rem;--bs-spinner-height:2rem;--bs-spinner-vertical-align:-0.125em;--bs-spinner-animation-speed:0.75s;--bs-spinner-animation-name:spinner-grow;background-color:currentcolor;opacity:0}.spinner-grow-sm{--bs-spinner-width:1rem;--bs-spinner-height:1rem}@media (prefers-reduced-motion:reduce){.spinner-border,.spinner-grow{--bs-spinner-animation-speed:1.5s}}.offcanvas,.offcanvas-lg,.offcanvas-md,.offcanvas-sm,.offcanvas-xl,.offcanvas-xxl{--bs-offcanvas-zindex:1045;--bs-offcanvas-width:400px;--bs-offcanvas-height:30vh;--bs-offcanvas-padding-x:1rem;--bs-offcanvas-padding-y:1rem;--bs-offcanvas-color:var(--bs-body-color);--bs-offcanvas-bg:var(--bs-body-bg);--bs-offcanvas-border-width:var(--bs-border-width);--bs-offcanvas-border-color:var(--bs-border-color-translucent);--bs-offcanvas-box-shadow:var(--bs-box-shadow-sm);--bs-offcanvas-transition:transform 0.3s ease-in-out;--bs-offcanvas-title-line-height:1.5}@media (max-width:575.98px){.offcanvas-sm{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}}@media (max-width:575.98px) and (prefers-reduced-motion:reduce){.offcanvas-sm{transition:none}}@media (max-width:575.98px){.offcanvas-sm.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas-sm.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas-sm.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas-sm.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas-sm.show:not(.hiding),.offcanvas-sm.showing{transform:none}.offcanvas-sm.hiding,.offcanvas-sm.show,.offcanvas-sm.showing{visibility:visible}}@media (min-width:576px){.offcanvas-sm{--bs-offcanvas-height:auto;--bs-offcanvas-border-width:0;background-color:transparent!important}.offcanvas-sm .offcanvas-header{display:none}.offcanvas-sm .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible;background-color:transparent!important}}@media (max-width:767.98px){.offcanvas-md{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}}@media (max-width:767.98px) and (prefers-reduced-motion:reduce){.offcanvas-md{transition:none}}@media (max-width:767.98px){.offcanvas-md.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas-md.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas-md.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas-md.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas-md.show:not(.hiding),.offcanvas-md.showing{transform:none}.offcanvas-md.hiding,.offcanvas-md.show,.offcanvas-md.showing{visibility:visible}}@media (min-width:768px){.offcanvas-md{--bs-offcanvas-height:auto;--bs-offcanvas-border-width:0;background-color:transparent!important}.offcanvas-md .offcanvas-header{display:none}.offcanvas-md .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible;background-color:transparent!important}}@media (max-width:991.98px){.offcanvas-lg{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}}@media (max-width:991.98px) and (prefers-reduced-motion:reduce){.offcanvas-lg{transition:none}}@media (max-width:991.98px){.offcanvas-lg.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas-lg.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas-lg.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas-lg.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas-lg.show:not(.hiding),.offcanvas-lg.showing{transform:none}.offcanvas-lg.hiding,.offcanvas-lg.show,.offcanvas-lg.showing{visibility:visible}}@media (min-width:992px){.offcanvas-lg{--bs-offcanvas-height:auto;--bs-offcanvas-border-width:0;background-color:transparent!important}.offcanvas-lg .offcanvas-header{display:none}.offcanvas-lg .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible;background-color:transparent!important}}@media (max-width:1199.98px){.offcanvas-xl{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}}@media (max-width:1199.98px) and (prefers-reduced-motion:reduce){.offcanvas-xl{transition:none}}@media (max-width:1199.98px){.offcanvas-xl.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas-xl.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas-xl.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas-xl.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas-xl.show:not(.hiding),.offcanvas-xl.showing{transform:none}.offcanvas-xl.hiding,.offcanvas-xl.show,.offcanvas-xl.showing{visibility:visible}}@media (min-width:1200px){.offcanvas-xl{--bs-offcanvas-height:auto;--bs-offcanvas-border-width:0;background-color:transparent!important}.offcanvas-xl .offcanvas-header{display:none}.offcanvas-xl .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible;background-color:transparent!important}}@media (max-width:1399.98px){.offcanvas-xxl{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}}@media (max-width:1399.98px) and (prefers-reduced-motion:reduce){.offcanvas-xxl{transition:none}}@media (max-width:1399.98px){.offcanvas-xxl.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas-xxl.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas-xxl.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas-xxl.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas-xxl.show:not(.hiding),.offcanvas-xxl.showing{transform:none}.offcanvas-xxl.hiding,.offcanvas-xxl.show,.offcanvas-xxl.showing{visibility:visible}}@media (min-width:1400px){.offcanvas-xxl{--bs-offcanvas-height:auto;--bs-offcanvas-border-width:0;background-color:transparent!important}.offcanvas-xxl .offcanvas-header{display:none}.offcanvas-xxl .offcanvas-body{display:flex;flex-grow:0;padding:0;overflow-y:visible;background-color:transparent!important}}.offcanvas{position:fixed;bottom:0;z-index:var(--bs-offcanvas-zindex);display:flex;flex-direction:column;max-width:100%;color:var(--bs-offcanvas-color);visibility:hidden;background-color:var(--bs-offcanvas-bg);background-clip:padding-box;outline:0;transition:var(--bs-offcanvas-transition)}@media (prefers-reduced-motion:reduce){.offcanvas{transition:none}}.offcanvas.offcanvas-start{top:0;left:0;width:var(--bs-offcanvas-width);border-right:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(-100%)}.offcanvas.offcanvas-end{top:0;right:0;width:var(--bs-offcanvas-width);border-left:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateX(100%)}.offcanvas.offcanvas-top{top:0;right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-bottom:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(-100%)}.offcanvas.offcanvas-bottom{right:0;left:0;height:var(--bs-offcanvas-height);max-height:100%;border-top:var(--bs-offcanvas-border-width) solid var(--bs-offcanvas-border-color);transform:translateY(100%)}.offcanvas.show:not(.hiding),.offcanvas.showing{transform:none}.offcanvas.hiding,.offcanvas.show,.offcanvas.showing{visibility:visible}.offcanvas-backdrop{position:fixed;top:0;left:0;z-index:1040;width:100vw;height:100vh;background-color:#000}.offcanvas-backdrop.fade{opacity:0}.offcanvas-backdrop.show{opacity:.5}.offcanvas-header{display:flex;align-items:center;padding:var(--bs-offcanvas-padding-y) var(--bs-offcanvas-padding-x)}.offcanvas-header .btn-close{padding:calc(var(--bs-offcanvas-padding-y) * .5) calc(var(--bs-offcanvas-padding-x) * .5);margin:calc(-.5 * var(--bs-offcanvas-padding-y)) calc(-.5 * var(--bs-offcanvas-padding-x)) calc(-.5 * var(--bs-offcanvas-padding-y)) auto}.offcanvas-title{margin-bottom:0;line-height:var(--bs-offcanvas-title-line-height)}.offcanvas-body{flex-grow:1;padding:var(--bs-offcanvas-padding-y) var(--bs-offcanvas-padding-x);overflow-y:auto}.placeholder{display:inline-block;min-height:1em;vertical-align:middle;cursor:wait;background-color:currentcolor;opacity:.5}.placeholder.btn::before{display:inline-block;content:""}.placeholder-xs{min-height:.6em}.placeholder-sm{min-height:.8em}.placeholder-lg{min-height:1.2em}.placeholder-glow .placeholder{animation:placeholder-glow 2s ease-in-out infinite}@keyframes placeholder-glow{50%{opacity:.2}}.placeholder-wave{-webkit-mask-image:linear-gradient(130deg,#000 55%,rgba(0,0,0,0.8) 75%,#000 95%);mask-image:linear-gradient(130deg,#000 55%,rgba(0,0,0,0.8) 75%,#000 95%);-webkit-mask-size:200% 100%;mask-size:200% 100%;animation:placeholder-wave 2s linear infinite}@keyframes placeholder-wave{100%{-webkit-mask-position:-200% 0%;mask-position:-200% 0%}}.clearfix::after{display:block;clear:both;content:""}.text-bg-primary{color:#fff!important;background-color:RGBA(var(--bs-primary-rgb),var(--bs-bg-opacity,1))!important}.text-bg-secondary{color:#fff!important;background-color:RGBA(var(--bs-secondary-rgb),var(--bs-bg-opacity,1))!important}.text-bg-success{color:#fff!important;background-color:RGBA(var(--bs-success-rgb),var(--bs-bg-opacity,1))!important}.text-bg-info{color:#000!important;background-color:RGBA(var(--bs-info-rgb),var(--bs-bg-opacity,1))!important}.text-bg-warning{color:#000!important;background-color:RGBA(var(--bs-warning-rgb),var(--bs-bg-opacity,1))!important}.text-bg-danger{color:#fff!important;background-color:RGBA(var(--bs-danger-rgb),var(--bs-bg-opacity,1))!important}.text-bg-light{color:#000!important;background-color:RGBA(var(--bs-light-rgb),var(--bs-bg-opacity,1))!important}.text-bg-dark{color:#fff!important;background-color:RGBA(var(--bs-dark-rgb),var(--bs-bg-opacity,1))!important}.link-primary{color:RGBA(var(--bs-primary-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-primary-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-primary-rgb),var(--bs-link-underline-opacity,1))!important}.link-primary:focus,.link-primary:hover{color:RGBA(10,88,202,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(10,88,202,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(10,88,202,var(--bs-link-underline-opacity,1))!important}.link-secondary{color:RGBA(var(--bs-secondary-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-secondary-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-secondary-rgb),var(--bs-link-underline-opacity,1))!important}.link-secondary:focus,.link-secondary:hover{color:RGBA(86,94,100,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(86,94,100,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(86,94,100,var(--bs-link-underline-opacity,1))!important}.link-success{color:RGBA(var(--bs-success-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-success-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-success-rgb),var(--bs-link-underline-opacity,1))!important}.link-success:focus,.link-success:hover{color:RGBA(20,108,67,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(20,108,67,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(20,108,67,var(--bs-link-underline-opacity,1))!important}.link-info{color:RGBA(var(--bs-info-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-info-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-info-rgb),var(--bs-link-underline-opacity,1))!important}.link-info:focus,.link-info:hover{color:RGBA(61,213,243,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(61,213,243,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(61,213,243,var(--bs-link-underline-opacity,1))!important}.link-warning{color:RGBA(var(--bs-warning-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-warning-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-warning-rgb),var(--bs-link-underline-opacity,1))!important}.link-warning:focus,.link-warning:hover{color:RGBA(255,205,57,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(255,205,57,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(255,205,57,var(--bs-link-underline-opacity,1))!important}.link-danger{color:RGBA(var(--bs-danger-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-danger-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-danger-rgb),var(--bs-link-underline-opacity,1))!important}.link-danger:focus,.link-danger:hover{color:RGBA(176,42,55,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(176,42,55,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(176,42,55,var(--bs-link-underline-opacity,1))!important}.link-light{color:RGBA(var(--bs-light-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-light-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-light-rgb),var(--bs-link-underline-opacity,1))!important}.link-light:focus,.link-light:hover{color:RGBA(249,250,251,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(249,250,251,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(249,250,251,var(--bs-link-underline-opacity,1))!important}.link-dark{color:RGBA(var(--bs-dark-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-dark-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-dark-rgb),var(--bs-link-underline-opacity,1))!important}.link-dark:focus,.link-dark:hover{color:RGBA(26,30,33,var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(26,30,33,var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(26,30,33,var(--bs-link-underline-opacity,1))!important}.link-body-emphasis{color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-opacity,1))!important;-webkit-text-decoration-color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-underline-opacity,1))!important}.link-body-emphasis:focus,.link-body-emphasis:hover{color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-opacity,.75))!important;-webkit-text-decoration-color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-underline-opacity,0.75))!important;text-decoration-color:RGBA(var(--bs-emphasis-color-rgb),var(--bs-link-underline-opacity,0.75))!important}.focus-ring:focus{outline:0;box-shadow:var(--bs-focus-ring-x,0) var(--bs-focus-ring-y,0) var(--bs-focus-ring-blur,0) var(--bs-focus-ring-width) var(--bs-focus-ring-color)}.icon-link{display:inline-flex;gap:.375rem;align-items:center;-webkit-text-decoration-color:rgba(var(--bs-link-color-rgb),var(--bs-link-opacity,0.5));text-decoration-color:rgba(var(--bs-link-color-rgb),var(--bs-link-opacity,0.5));text-underline-offset:0.25em;-webkit-backface-visibility:hidden;backface-visibility:hidden}.icon-link>.bi{flex-shrink:0;width:1em;height:1em;fill:currentcolor;transition:.2s ease-in-out transform}@media (prefers-reduced-motion:reduce){.icon-link>.bi{transition:none}}.icon-link-hover:focus-visible>.bi,.icon-link-hover:hover>.bi{transform:var(--bs-icon-link-transform,translate3d(.25em,0,0))}.ratio{position:relative;width:100%}.ratio::before{display:block;padding-top:var(--bs-aspect-ratio);content:""}.ratio>*{position:absolute;top:0;left:0;width:100%;height:100%}.ratio-1x1{--bs-aspect-ratio:100%}.ratio-4x3{--bs-aspect-ratio:75%}.ratio-16x9{--bs-aspect-ratio:56.25%}.ratio-21x9{--bs-aspect-ratio:42.8571428571%}.fixed-top{position:fixed;top:0;right:0;left:0;z-index:1030}.fixed-bottom{position:fixed;right:0;bottom:0;left:0;z-index:1030}.sticky-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}@media (min-width:576px){.sticky-sm-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-sm-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}}@media (min-width:768px){.sticky-md-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-md-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}}@media (min-width:992px){.sticky-lg-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-lg-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}}@media (min-width:1200px){.sticky-xl-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-xl-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}}@media (min-width:1400px){.sticky-xxl-top{position:-webkit-sticky;position:sticky;top:0;z-index:1020}.sticky-xxl-bottom{position:-webkit-sticky;position:sticky;bottom:0;z-index:1020}}.hstack{display:flex;flex-direction:row;align-items:center;align-self:stretch}.vstack{display:flex;flex:1 1 auto;flex-direction:column;align-self:stretch}.visually-hidden,.visually-hidden-focusable:not(:focus):not(:focus-within){width:1px!important;height:1px!important;padding:0!important;margin:-1px!important;overflow:hidden!important;clip:rect(0,0,0,0)!important;white-space:nowrap!important;border:0!important}.visually-hidden-focusable:not(:focus):not(:focus-within):not(caption),.visually-hidden:not(caption){position:absolute!important}.stretched-link::after{position:absolute;top:0;right:0;bottom:0;left:0;z-index:1;content:""}.text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.vr{display:inline-block;align-self:stretch;width:var(--bs-border-width);min-height:1em;background-color:currentcolor;opacity:.25}.align-baseline{vertical-align:baseline!important}.align-top{vertical-align:top!important}.align-middle{vertical-align:middle!important}.align-bottom{vertical-align:bottom!important}.align-text-bottom{vertical-align:text-bottom!important}.align-text-top{vertical-align:text-top!important}.float-start{float:left!important}.float-end{float:right!important}.float-none{float:none!important}.object-fit-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-none{-o-object-fit:none!important;object-fit:none!important}.opacity-0{opacity:0!important}.opacity-25{opacity:.25!important}.opacity-50{opacity:.5!important}.opacity-75{opacity:.75!important}.opacity-100{opacity:1!important}.overflow-auto{overflow:auto!important}.overflow-hidden{overflow:hidden!important}.overflow-visible{overflow:visible!important}.overflow-scroll{overflow:scroll!important}.overflow-x-auto{overflow-x:auto!important}.overflow-x-hidden{overflow-x:hidden!important}.overflow-x-visible{overflow-x:visible!important}.overflow-x-scroll{overflow-x:scroll!important}.overflow-y-auto{overflow-y:auto!important}.overflow-y-hidden{overflow-y:hidden!important}.overflow-y-visible{overflow-y:visible!important}.overflow-y-scroll{overflow-y:scroll!important}.d-inline{display:inline!important}.d-inline-block{display:inline-block!important}.d-block{display:block!important}.d-grid{display:grid!important}.d-inline-grid{display:inline-grid!important}.d-table{display:table!important}.d-table-row{display:table-row!important}.d-table-cell{display:table-cell!important}.d-flex{display:flex!important}.d-inline-flex{display:inline-flex!important}.d-none{display:none!important}.shadow{box-shadow:var(--bs-box-shadow)!important}.shadow-sm{box-shadow:var(--bs-box-shadow-sm)!important}.shadow-lg{box-shadow:var(--bs-box-shadow-lg)!important}.shadow-none{box-shadow:none!important}.focus-ring-primary{--bs-focus-ring-color:rgba(var(--bs-primary-rgb), var(--bs-focus-ring-opacity))}.focus-ring-secondary{--bs-focus-ring-color:rgba(var(--bs-secondary-rgb), var(--bs-focus-ring-opacity))}.focus-ring-success{--bs-focus-ring-color:rgba(var(--bs-success-rgb), var(--bs-focus-ring-opacity))}.focus-ring-info{--bs-focus-ring-color:rgba(var(--bs-info-rgb), var(--bs-focus-ring-opacity))}.focus-ring-warning{--bs-focus-ring-color:rgba(var(--bs-warning-rgb), var(--bs-focus-ring-opacity))}.focus-ring-danger{--bs-focus-ring-color:rgba(var(--bs-danger-rgb), var(--bs-focus-ring-opacity))}.focus-ring-light{--bs-focus-ring-color:rgba(var(--bs-light-rgb), var(--bs-focus-ring-opacity))}.focus-ring-dark{--bs-focus-ring-color:rgba(var(--bs-dark-rgb), var(--bs-focus-ring-opacity))}.position-static{position:static!important}.position-relative{position:relative!important}.position-absolute{position:absolute!important}.position-fixed{position:fixed!important}.position-sticky{position:-webkit-sticky!important;position:sticky!important}.top-0{top:0!important}.top-50{top:50%!important}.top-100{top:100%!important}.bottom-0{bottom:0!important}.bottom-50{bottom:50%!important}.bottom-100{bottom:100%!important}.start-0{left:0!important}.start-50{left:50%!important}.start-100{left:100%!important}.end-0{right:0!important}.end-50{right:50%!important}.end-100{right:100%!important}.translate-middle{transform:translate(-50%,-50%)!important}.translate-middle-x{transform:translateX(-50%)!important}.translate-middle-y{transform:translateY(-50%)!important}.border{border:var(--bs-border-width) var(--bs-border-style) var(--bs-border-color)!important}.border-0{border:0!important}.border-top{border-top:var(--bs-border-width) var(--bs-border-style) var(--bs-border-color)!important}.border-top-0{border-top:0!important}.border-end{border-right:var(--bs-border-width) var(--bs-border-style) var(--bs-border-color)!important}.border-end-0{border-right:0!important}.border-bottom{border-bottom:var(--bs-border-width) var(--bs-border-style) var(--bs-border-color)!important}.border-bottom-0{border-bottom:0!important}.border-start{border-left:var(--bs-border-width) var(--bs-border-style) var(--bs-border-color)!important}.border-start-0{border-left:0!important}.border-primary{--bs-border-opacity:1;border-color:rgba(var(--bs-primary-rgb),var(--bs-border-opacity))!important}.border-secondary{--bs-border-opacity:1;border-color:rgba(var(--bs-secondary-rgb),var(--bs-border-opacity))!important}.border-success{--bs-border-opacity:1;border-color:rgba(var(--bs-success-rgb),var(--bs-border-opacity))!important}.border-info{--bs-border-opacity:1;border-color:rgba(var(--bs-info-rgb),var(--bs-border-opacity))!important}.border-warning{--bs-border-opacity:1;border-color:rgba(var(--bs-warning-rgb),var(--bs-border-opacity))!important}.border-danger{--bs-border-opacity:1;border-color:rgba(var(--bs-danger-rgb),var(--bs-border-opacity))!important}.border-light{--bs-border-opacity:1;border-color:rgba(var(--bs-light-rgb),var(--bs-border-opacity))!important}.border-dark{--bs-border-opacity:1;border-color:rgba(var(--bs-dark-rgb),var(--bs-border-opacity))!important}.border-black{--bs-border-opacity:1;border-color:rgba(var(--bs-black-rgb),var(--bs-border-opacity))!important}.border-white{--bs-border-opacity:1;border-color:rgba(var(--bs-white-rgb),var(--bs-border-opacity))!important}.border-primary-subtle{border-color:var(--bs-primary-border-subtle)!important}.border-secondary-subtle{border-color:var(--bs-secondary-border-subtle)!important}.border-success-subtle{border-color:var(--bs-success-border-subtle)!important}.border-info-subtle{border-color:var(--bs-info-border-subtle)!important}.border-warning-subtle{border-color:var(--bs-warning-border-subtle)!important}.border-danger-subtle{border-color:var(--bs-danger-border-subtle)!important}.border-light-subtle{border-color:var(--bs-light-border-subtle)!important}.border-dark-subtle{border-color:var(--bs-dark-border-subtle)!important}.border-1{border-width:1px!important}.border-2{border-width:2px!important}.border-3{border-width:3px!important}.border-4{border-width:4px!important}.border-5{border-width:5px!important}.border-opacity-10{--bs-border-opacity:0.1}.border-opacity-25{--bs-border-opacity:0.25}.border-opacity-50{--bs-border-opacity:0.5}.border-opacity-75{--bs-border-opacity:0.75}.border-opacity-100{--bs-border-opacity:1}.w-25{width:25%!important}.w-50{width:50%!important}.w-75{width:75%!important}.w-100{width:100%!important}.w-auto{width:auto!important}.mw-100{max-width:100%!important}.vw-100{width:100vw!important}.min-vw-100{min-width:100vw!important}.h-25{height:25%!important}.h-50{height:50%!important}.h-75{height:75%!important}.h-100{height:100%!important}.h-auto{height:auto!important}.mh-100{max-height:100%!important}.vh-100{height:100vh!important}.min-vh-100{min-height:100vh!important}.flex-fill{flex:1 1 auto!important}.flex-row{flex-direction:row!important}.flex-column{flex-direction:column!important}.flex-row-reverse{flex-direction:row-reverse!important}.flex-column-reverse{flex-direction:column-reverse!important}.flex-grow-0{flex-grow:0!important}.flex-grow-1{flex-grow:1!important}.flex-shrink-0{flex-shrink:0!important}.flex-shrink-1{flex-shrink:1!important}.flex-wrap{flex-wrap:wrap!important}.flex-nowrap{flex-wrap:nowrap!important}.flex-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-start{justify-content:flex-start!important}.justify-content-end{justify-content:flex-end!important}.justify-content-center{justify-content:center!important}.justify-content-between{justify-content:space-between!important}.justify-content-around{justify-content:space-around!important}.justify-content-evenly{justify-content:space-evenly!important}.align-items-start{align-items:flex-start!important}.align-items-end{align-items:flex-end!important}.align-items-center{align-items:center!important}.align-items-baseline{align-items:baseline!important}.align-items-stretch{align-items:stretch!important}.align-content-start{align-content:flex-start!important}.align-content-end{align-content:flex-end!important}.align-content-center{align-content:center!important}.align-content-between{align-content:space-between!important}.align-content-around{align-content:space-around!important}.align-content-stretch{align-content:stretch!important}.align-self-auto{align-self:auto!important}.align-self-start{align-self:flex-start!important}.align-self-end{align-self:flex-end!important}.align-self-center{align-self:center!important}.align-self-baseline{align-self:baseline!important}.align-self-stretch{align-self:stretch!important}.order-first{order:-1!important}.order-0{order:0!important}.order-1{order:1!important}.order-2{order:2!important}.order-3{order:3!important}.order-4{order:4!important}.order-5{order:5!important}.order-last{order:6!important}.m-0{margin:0!important}.m-1{margin:.25rem!important}.m-2{margin:.5rem!important}.m-3{margin:1rem!important}.m-4{margin:1.5rem!important}.m-5{margin:3rem!important}.m-auto{margin:auto!important}.mx-0{margin-right:0!important;margin-left:0!important}.mx-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-3{margin-right:1rem!important;margin-left:1rem!important}.mx-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-5{margin-right:3rem!important;margin-left:3rem!important}.mx-auto{margin-right:auto!important;margin-left:auto!important}.my-0{margin-top:0!important;margin-bottom:0!important}.my-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-0{margin-top:0!important}.mt-1{margin-top:.25rem!important}.mt-2{margin-top:.5rem!important}.mt-3{margin-top:1rem!important}.mt-4{margin-top:1.5rem!important}.mt-5{margin-top:3rem!important}.mt-auto{margin-top:auto!important}.me-0{margin-right:0!important}.me-1{margin-right:.25rem!important}.me-2{margin-right:.5rem!important}.me-3{margin-right:1rem!important}.me-4{margin-right:1.5rem!important}.me-5{margin-right:3rem!important}.me-auto{margin-right:auto!important}.mb-0{margin-bottom:0!important}.mb-1{margin-bottom:.25rem!important}.mb-2{margin-bottom:.5rem!important}.mb-3{margin-bottom:1rem!important}.mb-4{margin-bottom:1.5rem!important}.mb-5{margin-bottom:3rem!important}.mb-auto{margin-bottom:auto!important}.ms-0{margin-left:0!important}.ms-1{margin-left:.25rem!important}.ms-2{margin-left:.5rem!important}.ms-3{margin-left:1rem!important}.ms-4{margin-left:1.5rem!important}.ms-5{margin-left:3rem!important}.ms-auto{margin-left:auto!important}.p-0{padding:0!important}.p-1{padding:.25rem!important}.p-2{padding:.5rem!important}.p-3{padding:1rem!important}.p-4{padding:1.5rem!important}.p-5{padding:3rem!important}.px-0{padding-right:0!important;padding-left:0!important}.px-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-3{padding-right:1rem!important;padding-left:1rem!important}.px-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-5{padding-right:3rem!important;padding-left:3rem!important}.py-0{padding-top:0!important;padding-bottom:0!important}.py-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-0{padding-top:0!important}.pt-1{padding-top:.25rem!important}.pt-2{padding-top:.5rem!important}.pt-3{padding-top:1rem!important}.pt-4{padding-top:1.5rem!important}.pt-5{padding-top:3rem!important}.pe-0{padding-right:0!important}.pe-1{padding-right:.25rem!important}.pe-2{padding-right:.5rem!important}.pe-3{padding-right:1rem!important}.pe-4{padding-right:1.5rem!important}.pe-5{padding-right:3rem!important}.pb-0{padding-bottom:0!important}.pb-1{padding-bottom:.25rem!important}.pb-2{padding-bottom:.5rem!important}.pb-3{padding-bottom:1rem!important}.pb-4{padding-bottom:1.5rem!important}.pb-5{padding-bottom:3rem!important}.ps-0{padding-left:0!important}.ps-1{padding-left:.25rem!important}.ps-2{padding-left:.5rem!important}.ps-3{padding-left:1rem!important}.ps-4{padding-left:1.5rem!important}.ps-5{padding-left:3rem!important}.gap-0{gap:0!important}.gap-1{gap:.25rem!important}.gap-2{gap:.5rem!important}.gap-3{gap:1rem!important}.gap-4{gap:1.5rem!important}.gap-5{gap:3rem!important}.row-gap-0{row-gap:0!important}.row-gap-1{row-gap:.25rem!important}.row-gap-2{row-gap:.5rem!important}.row-gap-3{row-gap:1rem!important}.row-gap-4{row-gap:1.5rem!important}.row-gap-5{row-gap:3rem!important}.column-gap-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.font-monospace{font-family:var(--bs-font-monospace)!important}.fs-1{font-size:calc(1.375rem + 1.5vw)!important}.fs-2{font-size:calc(1.325rem + .9vw)!important}.fs-3{font-size:calc(1.3rem + .6vw)!important}.fs-4{font-size:calc(1.275rem + .3vw)!important}.fs-5{font-size:1.25rem!important}.fs-6{font-size:1rem!important}.fst-italic{font-style:italic!important}.fst-normal{font-style:normal!important}.fw-lighter{font-weight:lighter!important}.fw-light{font-weight:300!important}.fw-normal{font-weight:400!important}.fw-medium{font-weight:500!important}.fw-semibold{font-weight:600!important}.fw-bold{font-weight:700!important}.fw-bolder{font-weight:bolder!important}.lh-1{line-height:1!important}.lh-sm{line-height:1.25!important}.lh-base{line-height:1.5!important}.lh-lg{line-height:2!important}.text-start{text-align:left!important}.text-end{text-align:right!important}.text-center{text-align:center!important}.text-decoration-none{text-decoration:none!important}.text-decoration-underline{text-decoration:underline!important}.text-decoration-line-through{text-decoration:line-through!important}.text-lowercase{text-transform:lowercase!important}.text-uppercase{text-transform:uppercase!important}.text-capitalize{text-transform:capitalize!important}.text-wrap{white-space:normal!important}.text-nowrap{white-space:nowrap!important}.text-break{word-wrap:break-word!important;word-break:break-word!important}.text-primary{--bs-text-opacity:1;color:rgba(var(--bs-primary-rgb),var(--bs-text-opacity))!important}.text-secondary{--bs-text-opacity:1;color:rgba(var(--bs-secondary-rgb),var(--bs-text-opacity))!important}.text-success{--bs-text-opacity:1;color:rgba(var(--bs-success-rgb),var(--bs-text-opacity))!important}.text-info{--bs-text-opacity:1;color:rgba(var(--bs-info-rgb),var(--bs-text-opacity))!important}.text-warning{--bs-text-opacity:1;color:rgba(var(--bs-warning-rgb),var(--bs-text-opacity))!important}.text-danger{--bs-text-opacity:1;color:rgba(var(--bs-danger-rgb),var(--bs-text-opacity))!important}.text-light{--bs-text-opacity:1;color:rgba(var(--bs-light-rgb),var(--bs-text-opacity))!important}.text-dark{--bs-text-opacity:1;color:rgba(var(--bs-dark-rgb),var(--bs-text-opacity))!important}.text-black{--bs-text-opacity:1;color:rgba(var(--bs-black-rgb),var(--bs-text-opacity))!important}.text-white{--bs-text-opacity:1;color:rgba(var(--bs-white-rgb),var(--bs-text-opacity))!important}.text-body{--bs-text-opacity:1;color:rgba(var(--bs-body-color-rgb),var(--bs-text-opacity))!important}.text-muted{--bs-text-opacity:1;color:var(--bs-secondary-color)!important}.text-black-50{--bs-text-opacity:1;color:rgba(0,0,0,.5)!important}.text-white-50{--bs-text-opacity:1;color:rgba(255,255,255,.5)!important}.text-body-secondary{--bs-text-opacity:1;color:var(--bs-secondary-color)!important}.text-body-tertiary{--bs-text-opacity:1;color:var(--bs-tertiary-color)!important}.text-body-emphasis{--bs-text-opacity:1;color:var(--bs-emphasis-color)!important}.text-reset{--bs-text-opacity:1;color:inherit!important}.text-opacity-25{--bs-text-opacity:0.25}.text-opacity-50{--bs-text-opacity:0.5}.text-opacity-75{--bs-text-opacity:0.75}.text-opacity-100{--bs-text-opacity:1}.text-primary-emphasis{color:var(--bs-primary-text-emphasis)!important}.text-secondary-emphasis{color:var(--bs-secondary-text-emphasis)!important}.text-success-emphasis{color:var(--bs-success-text-emphasis)!important}.text-info-emphasis{color:var(--bs-info-text-emphasis)!important}.text-warning-emphasis{color:var(--bs-warning-text-emphasis)!important}.text-danger-emphasis{color:var(--bs-danger-text-emphasis)!important}.text-light-emphasis{color:var(--bs-light-text-emphasis)!important}.text-dark-emphasis{color:var(--bs-dark-text-emphasis)!important}.link-opacity-10{--bs-link-opacity:0.1}.link-opacity-10-hover:hover{--bs-link-opacity:0.1}.link-opacity-25{--bs-link-opacity:0.25}.link-opacity-25-hover:hover{--bs-link-opacity:0.25}.link-opacity-50{--bs-link-opacity:0.5}.link-opacity-50-hover:hover{--bs-link-opacity:0.5}.link-opacity-75{--bs-link-opacity:0.75}.link-opacity-75-hover:hover{--bs-link-opacity:0.75}.link-opacity-100{--bs-link-opacity:1}.link-opacity-100-hover:hover{--bs-link-opacity:1}.link-offset-1{text-underline-offset:0.125em!important}.link-offset-1-hover:hover{text-underline-offset:0.125em!important}.link-offset-2{text-underline-offset:0.25em!important}.link-offset-2-hover:hover{text-underline-offset:0.25em!important}.link-offset-3{text-underline-offset:0.375em!important}.link-offset-3-hover:hover{text-underline-offset:0.375em!important}.link-underline-primary{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-primary-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-primary-rgb),var(--bs-link-underline-opacity))!important}.link-underline-secondary{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-secondary-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-secondary-rgb),var(--bs-link-underline-opacity))!important}.link-underline-success{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-success-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-success-rgb),var(--bs-link-underline-opacity))!important}.link-underline-info{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-info-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-info-rgb),var(--bs-link-underline-opacity))!important}.link-underline-warning{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-warning-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-warning-rgb),var(--bs-link-underline-opacity))!important}.link-underline-danger{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-danger-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-danger-rgb),var(--bs-link-underline-opacity))!important}.link-underline-light{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-light-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-light-rgb),var(--bs-link-underline-opacity))!important}.link-underline-dark{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-dark-rgb),var(--bs-link-underline-opacity))!important;text-decoration-color:rgba(var(--bs-dark-rgb),var(--bs-link-underline-opacity))!important}.link-underline{--bs-link-underline-opacity:1;-webkit-text-decoration-color:rgba(var(--bs-link-color-rgb),var(--bs-link-underline-opacity,1))!important;text-decoration-color:rgba(var(--bs-link-color-rgb),var(--bs-link-underline-opacity,1))!important}.link-underline-opacity-0{--bs-link-underline-opacity:0}.link-underline-opacity-0-hover:hover{--bs-link-underline-opacity:0}.link-underline-opacity-10{--bs-link-underline-opacity:0.1}.link-underline-opacity-10-hover:hover{--bs-link-underline-opacity:0.1}.link-underline-opacity-25{--bs-link-underline-opacity:0.25}.link-underline-opacity-25-hover:hover{--bs-link-underline-opacity:0.25}.link-underline-opacity-50{--bs-link-underline-opacity:0.5}.link-underline-opacity-50-hover:hover{--bs-link-underline-opacity:0.5}.link-underline-opacity-75{--bs-link-underline-opacity:0.75}.link-underline-opacity-75-hover:hover{--bs-link-underline-opacity:0.75}.link-underline-opacity-100{--bs-link-underline-opacity:1}.link-underline-opacity-100-hover:hover{--bs-link-underline-opacity:1}.bg-primary{--bs-bg-opacity:1;background-color:rgba(var(--bs-primary-rgb),var(--bs-bg-opacity))!important}.bg-secondary{--bs-bg-opacity:1;background-color:rgba(var(--bs-secondary-rgb),var(--bs-bg-opacity))!important}.bg-success{--bs-bg-opacity:1;background-color:rgba(var(--bs-success-rgb),var(--bs-bg-opacity))!important}.bg-info{--bs-bg-opacity:1;background-color:rgba(var(--bs-info-rgb),var(--bs-bg-opacity))!important}.bg-warning{--bs-bg-opacity:1;background-color:rgba(var(--bs-warning-rgb),var(--bs-bg-opacity))!important}.bg-danger{--bs-bg-opacity:1;background-color:rgba(var(--bs-danger-rgb),var(--bs-bg-opacity))!important}.bg-light{--bs-bg-opacity:1;background-color:rgba(var(--bs-light-rgb),var(--bs-bg-opacity))!important}.bg-dark{--bs-bg-opacity:1;background-color:rgba(var(--bs-dark-rgb),var(--bs-bg-opacity))!important}.bg-black{--bs-bg-opacity:1;background-color:rgba(var(--bs-black-rgb),var(--bs-bg-opacity))!important}.bg-white{--bs-bg-opacity:1;background-color:rgba(var(--bs-white-rgb),var(--bs-bg-opacity))!important}.bg-body{--bs-bg-opacity:1;background-color:rgba(var(--bs-body-bg-rgb),var(--bs-bg-opacity))!important}.bg-transparent{--bs-bg-opacity:1;background-color:transparent!important}.bg-body-secondary{--bs-bg-opacity:1;background-color:rgba(var(--bs-secondary-bg-rgb),var(--bs-bg-opacity))!important}.bg-body-tertiary{--bs-bg-opacity:1;background-color:rgba(var(--bs-tertiary-bg-rgb),var(--bs-bg-opacity))!important}.bg-opacity-10{--bs-bg-opacity:0.1}.bg-opacity-25{--bs-bg-opacity:0.25}.bg-opacity-50{--bs-bg-opacity:0.5}.bg-opacity-75{--bs-bg-opacity:0.75}.bg-opacity-100{--bs-bg-opacity:1}.bg-primary-subtle{background-color:var(--bs-primary-bg-subtle)!important}.bg-secondary-subtle{background-color:var(--bs-secondary-bg-subtle)!important}.bg-success-subtle{background-color:var(--bs-success-bg-subtle)!important}.bg-info-subtle{background-color:var(--bs-info-bg-subtle)!important}.bg-warning-subtle{background-color:var(--bs-warning-bg-subtle)!important}.bg-danger-subtle{background-color:var(--bs-danger-bg-subtle)!important}.bg-light-subtle{background-color:var(--bs-light-bg-subtle)!important}.bg-dark-subtle{background-color:var(--bs-dark-bg-subtle)!important}.bg-gradient{background-image:var(--bs-gradient)!important}.user-select-all{-webkit-user-select:all!important;-moz-user-select:all!important;user-select:all!important}.user-select-auto{-webkit-user-select:auto!important;-moz-user-select:auto!important;user-select:auto!important}.user-select-none{-webkit-user-select:none!important;-moz-user-select:none!important;user-select:none!important}.pe-none{pointer-events:none!important}.pe-auto{pointer-events:auto!important}.rounded{border-radius:var(--bs-border-radius)!important}.rounded-0{border-radius:0!important}.rounded-1{border-radius:var(--bs-border-radius-sm)!important}.rounded-2{border-radius:var(--bs-border-radius)!important}.rounded-3{border-radius:var(--bs-border-radius-lg)!important}.rounded-4{border-radius:var(--bs-border-radius-xl)!important}.rounded-5{border-radius:var(--bs-border-radius-xxl)!important}.rounded-circle{border-radius:50%!important}.rounded-pill{border-radius:var(--bs-border-radius-pill)!important}.rounded-top{border-top-left-radius:var(--bs-border-radius)!important;border-top-right-radius:var(--bs-border-radius)!important}.rounded-top-0{border-top-left-radius:0!important;border-top-right-radius:0!important}.rounded-top-1{border-top-left-radius:var(--bs-border-radius-sm)!important;border-top-right-radius:var(--bs-border-radius-sm)!important}.rounded-top-2{border-top-left-radius:var(--bs-border-radius)!important;border-top-right-radius:var(--bs-border-radius)!important}.rounded-top-3{border-top-left-radius:var(--bs-border-radius-lg)!important;border-top-right-radius:var(--bs-border-radius-lg)!important}.rounded-top-4{border-top-left-radius:var(--bs-border-radius-xl)!important;border-top-right-radius:var(--bs-border-radius-xl)!important}.rounded-top-5{border-top-left-radius:var(--bs-border-radius-xxl)!important;border-top-right-radius:var(--bs-border-radius-xxl)!important}.rounded-top-circle{border-top-left-radius:50%!important;border-top-right-radius:50%!important}.rounded-top-pill{border-top-left-radius:var(--bs-border-radius-pill)!important;border-top-right-radius:var(--bs-border-radius-pill)!important}.rounded-end{border-top-right-radius:var(--bs-border-radius)!important;border-bottom-right-radius:var(--bs-border-radius)!important}.rounded-end-0{border-top-right-radius:0!important;border-bottom-right-radius:0!important}.rounded-end-1{border-top-right-radius:var(--bs-border-radius-sm)!important;border-bottom-right-radius:var(--bs-border-radius-sm)!important}.rounded-end-2{border-top-right-radius:var(--bs-border-radius)!important;border-bottom-right-radius:var(--bs-border-radius)!important}.rounded-end-3{border-top-right-radius:var(--bs-border-radius-lg)!important;border-bottom-right-radius:var(--bs-border-radius-lg)!important}.rounded-end-4{border-top-right-radius:var(--bs-border-radius-xl)!important;border-bottom-right-radius:var(--bs-border-radius-xl)!important}.rounded-end-5{border-top-right-radius:var(--bs-border-radius-xxl)!important;border-bottom-right-radius:var(--bs-border-radius-xxl)!important}.rounded-end-circle{border-top-right-radius:50%!important;border-bottom-right-radius:50%!important}.rounded-end-pill{border-top-right-radius:var(--bs-border-radius-pill)!important;border-bottom-right-radius:var(--bs-border-radius-pill)!important}.rounded-bottom{border-bottom-right-radius:var(--bs-border-radius)!important;border-bottom-left-radius:var(--bs-border-radius)!important}.rounded-bottom-0{border-bottom-right-radius:0!important;border-bottom-left-radius:0!important}.rounded-bottom-1{border-bottom-right-radius:var(--bs-border-radius-sm)!important;border-bottom-left-radius:var(--bs-border-radius-sm)!important}.rounded-bottom-2{border-bottom-right-radius:var(--bs-border-radius)!important;border-bottom-left-radius:var(--bs-border-radius)!important}.rounded-bottom-3{border-bottom-right-radius:var(--bs-border-radius-lg)!important;border-bottom-left-radius:var(--bs-border-radius-lg)!important}.rounded-bottom-4{border-bottom-right-radius:var(--bs-border-radius-xl)!important;border-bottom-left-radius:var(--bs-border-radius-xl)!important}.rounded-bottom-5{border-bottom-right-radius:var(--bs-border-radius-xxl)!important;border-bottom-left-radius:var(--bs-border-radius-xxl)!important}.rounded-bottom-circle{border-bottom-right-radius:50%!important;border-bottom-left-radius:50%!important}.rounded-bottom-pill{border-bottom-right-radius:var(--bs-border-radius-pill)!important;border-bottom-left-radius:var(--bs-border-radius-pill)!important}.rounded-start{border-bottom-left-radius:var(--bs-border-radius)!important;border-top-left-radius:var(--bs-border-radius)!important}.rounded-start-0{border-bottom-left-radius:0!important;border-top-left-radius:0!important}.rounded-start-1{border-bottom-left-radius:var(--bs-border-radius-sm)!important;border-top-left-radius:var(--bs-border-radius-sm)!important}.rounded-start-2{border-bottom-left-radius:var(--bs-border-radius)!important;border-top-left-radius:var(--bs-border-radius)!important}.rounded-start-3{border-bottom-left-radius:var(--bs-border-radius-lg)!important;border-top-left-radius:var(--bs-border-radius-lg)!important}.rounded-start-4{border-bottom-left-radius:var(--bs-border-radius-xl)!important;border-top-left-radius:var(--bs-border-radius-xl)!important}.rounded-start-5{border-bottom-left-radius:var(--bs-border-radius-xxl)!important;border-top-left-radius:var(--bs-border-radius-xxl)!important}.rounded-start-circle{border-bottom-left-radius:50%!important;border-top-left-radius:50%!important}.rounded-start-pill{border-bottom-left-radius:var(--bs-border-radius-pill)!important;border-top-left-radius:var(--bs-border-radius-pill)!important}.visible{visibility:visible!important}.invisible{visibility:hidden!important}.z-n1{z-index:-1!important}.z-0{z-index:0!important}.z-1{z-index:1!important}.z-2{z-index:2!important}.z-3{z-index:3!important}@media (min-width:576px){.float-sm-start{float:left!important}.float-sm-end{float:right!important}.float-sm-none{float:none!important}.object-fit-sm-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-sm-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-sm-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-sm-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-sm-none{-o-object-fit:none!important;object-fit:none!important}.d-sm-inline{display:inline!important}.d-sm-inline-block{display:inline-block!important}.d-sm-block{display:block!important}.d-sm-grid{display:grid!important}.d-sm-inline-grid{display:inline-grid!important}.d-sm-table{display:table!important}.d-sm-table-row{display:table-row!important}.d-sm-table-cell{display:table-cell!important}.d-sm-flex{display:flex!important}.d-sm-inline-flex{display:inline-flex!important}.d-sm-none{display:none!important}.flex-sm-fill{flex:1 1 auto!important}.flex-sm-row{flex-direction:row!important}.flex-sm-column{flex-direction:column!important}.flex-sm-row-reverse{flex-direction:row-reverse!important}.flex-sm-column-reverse{flex-direction:column-reverse!important}.flex-sm-grow-0{flex-grow:0!important}.flex-sm-grow-1{flex-grow:1!important}.flex-sm-shrink-0{flex-shrink:0!important}.flex-sm-shrink-1{flex-shrink:1!important}.flex-sm-wrap{flex-wrap:wrap!important}.flex-sm-nowrap{flex-wrap:nowrap!important}.flex-sm-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-sm-start{justify-content:flex-start!important}.justify-content-sm-end{justify-content:flex-end!important}.justify-content-sm-center{justify-content:center!important}.justify-content-sm-between{justify-content:space-between!important}.justify-content-sm-around{justify-content:space-around!important}.justify-content-sm-evenly{justify-content:space-evenly!important}.align-items-sm-start{align-items:flex-start!important}.align-items-sm-end{align-items:flex-end!important}.align-items-sm-center{align-items:center!important}.align-items-sm-baseline{align-items:baseline!important}.align-items-sm-stretch{align-items:stretch!important}.align-content-sm-start{align-content:flex-start!important}.align-content-sm-end{align-content:flex-end!important}.align-content-sm-center{align-content:center!important}.align-content-sm-between{align-content:space-between!important}.align-content-sm-around{align-content:space-around!important}.align-content-sm-stretch{align-content:stretch!important}.align-self-sm-auto{align-self:auto!important}.align-self-sm-start{align-self:flex-start!important}.align-self-sm-end{align-self:flex-end!important}.align-self-sm-center{align-self:center!important}.align-self-sm-baseline{align-self:baseline!important}.align-self-sm-stretch{align-self:stretch!important}.order-sm-first{order:-1!important}.order-sm-0{order:0!important}.order-sm-1{order:1!important}.order-sm-2{order:2!important}.order-sm-3{order:3!important}.order-sm-4{order:4!important}.order-sm-5{order:5!important}.order-sm-last{order:6!important}.m-sm-0{margin:0!important}.m-sm-1{margin:.25rem!important}.m-sm-2{margin:.5rem!important}.m-sm-3{margin:1rem!important}.m-sm-4{margin:1.5rem!important}.m-sm-5{margin:3rem!important}.m-sm-auto{margin:auto!important}.mx-sm-0{margin-right:0!important;margin-left:0!important}.mx-sm-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-sm-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-sm-3{margin-right:1rem!important;margin-left:1rem!important}.mx-sm-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-sm-5{margin-right:3rem!important;margin-left:3rem!important}.mx-sm-auto{margin-right:auto!important;margin-left:auto!important}.my-sm-0{margin-top:0!important;margin-bottom:0!important}.my-sm-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-sm-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-sm-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-sm-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-sm-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-sm-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-sm-0{margin-top:0!important}.mt-sm-1{margin-top:.25rem!important}.mt-sm-2{margin-top:.5rem!important}.mt-sm-3{margin-top:1rem!important}.mt-sm-4{margin-top:1.5rem!important}.mt-sm-5{margin-top:3rem!important}.mt-sm-auto{margin-top:auto!important}.me-sm-0{margin-right:0!important}.me-sm-1{margin-right:.25rem!important}.me-sm-2{margin-right:.5rem!important}.me-sm-3{margin-right:1rem!important}.me-sm-4{margin-right:1.5rem!important}.me-sm-5{margin-right:3rem!important}.me-sm-auto{margin-right:auto!important}.mb-sm-0{margin-bottom:0!important}.mb-sm-1{margin-bottom:.25rem!important}.mb-sm-2{margin-bottom:.5rem!important}.mb-sm-3{margin-bottom:1rem!important}.mb-sm-4{margin-bottom:1.5rem!important}.mb-sm-5{margin-bottom:3rem!important}.mb-sm-auto{margin-bottom:auto!important}.ms-sm-0{margin-left:0!important}.ms-sm-1{margin-left:.25rem!important}.ms-sm-2{margin-left:.5rem!important}.ms-sm-3{margin-left:1rem!important}.ms-sm-4{margin-left:1.5rem!important}.ms-sm-5{margin-left:3rem!important}.ms-sm-auto{margin-left:auto!important}.p-sm-0{padding:0!important}.p-sm-1{padding:.25rem!important}.p-sm-2{padding:.5rem!important}.p-sm-3{padding:1rem!important}.p-sm-4{padding:1.5rem!important}.p-sm-5{padding:3rem!important}.px-sm-0{padding-right:0!important;padding-left:0!important}.px-sm-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-sm-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-sm-3{padding-right:1rem!important;padding-left:1rem!important}.px-sm-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-sm-5{padding-right:3rem!important;padding-left:3rem!important}.py-sm-0{padding-top:0!important;padding-bottom:0!important}.py-sm-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-sm-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-sm-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-sm-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-sm-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-sm-0{padding-top:0!important}.pt-sm-1{padding-top:.25rem!important}.pt-sm-2{padding-top:.5rem!important}.pt-sm-3{padding-top:1rem!important}.pt-sm-4{padding-top:1.5rem!important}.pt-sm-5{padding-top:3rem!important}.pe-sm-0{padding-right:0!important}.pe-sm-1{padding-right:.25rem!important}.pe-sm-2{padding-right:.5rem!important}.pe-sm-3{padding-right:1rem!important}.pe-sm-4{padding-right:1.5rem!important}.pe-sm-5{padding-right:3rem!important}.pb-sm-0{padding-bottom:0!important}.pb-sm-1{padding-bottom:.25rem!important}.pb-sm-2{padding-bottom:.5rem!important}.pb-sm-3{padding-bottom:1rem!important}.pb-sm-4{padding-bottom:1.5rem!important}.pb-sm-5{padding-bottom:3rem!important}.ps-sm-0{padding-left:0!important}.ps-sm-1{padding-left:.25rem!important}.ps-sm-2{padding-left:.5rem!important}.ps-sm-3{padding-left:1rem!important}.ps-sm-4{padding-left:1.5rem!important}.ps-sm-5{padding-left:3rem!important}.gap-sm-0{gap:0!important}.gap-sm-1{gap:.25rem!important}.gap-sm-2{gap:.5rem!important}.gap-sm-3{gap:1rem!important}.gap-sm-4{gap:1.5rem!important}.gap-sm-5{gap:3rem!important}.row-gap-sm-0{row-gap:0!important}.row-gap-sm-1{row-gap:.25rem!important}.row-gap-sm-2{row-gap:.5rem!important}.row-gap-sm-3{row-gap:1rem!important}.row-gap-sm-4{row-gap:1.5rem!important}.row-gap-sm-5{row-gap:3rem!important}.column-gap-sm-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-sm-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-sm-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-sm-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-sm-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-sm-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.text-sm-start{text-align:left!important}.text-sm-end{text-align:right!important}.text-sm-center{text-align:center!important}}@media (min-width:768px){.float-md-start{float:left!important}.float-md-end{float:right!important}.float-md-none{float:none!important}.object-fit-md-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-md-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-md-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-md-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-md-none{-o-object-fit:none!important;object-fit:none!important}.d-md-inline{display:inline!important}.d-md-inline-block{display:inline-block!important}.d-md-block{display:block!important}.d-md-grid{display:grid!important}.d-md-inline-grid{display:inline-grid!important}.d-md-table{display:table!important}.d-md-table-row{display:table-row!important}.d-md-table-cell{display:table-cell!important}.d-md-flex{display:flex!important}.d-md-inline-flex{display:inline-flex!important}.d-md-none{display:none!important}.flex-md-fill{flex:1 1 auto!important}.flex-md-row{flex-direction:row!important}.flex-md-column{flex-direction:column!important}.flex-md-row-reverse{flex-direction:row-reverse!important}.flex-md-column-reverse{flex-direction:column-reverse!important}.flex-md-grow-0{flex-grow:0!important}.flex-md-grow-1{flex-grow:1!important}.flex-md-shrink-0{flex-shrink:0!important}.flex-md-shrink-1{flex-shrink:1!important}.flex-md-wrap{flex-wrap:wrap!important}.flex-md-nowrap{flex-wrap:nowrap!important}.flex-md-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-md-start{justify-content:flex-start!important}.justify-content-md-end{justify-content:flex-end!important}.justify-content-md-center{justify-content:center!important}.justify-content-md-between{justify-content:space-between!important}.justify-content-md-around{justify-content:space-around!important}.justify-content-md-evenly{justify-content:space-evenly!important}.align-items-md-start{align-items:flex-start!important}.align-items-md-end{align-items:flex-end!important}.align-items-md-center{align-items:center!important}.align-items-md-baseline{align-items:baseline!important}.align-items-md-stretch{align-items:stretch!important}.align-content-md-start{align-content:flex-start!important}.align-content-md-end{align-content:flex-end!important}.align-content-md-center{align-content:center!important}.align-content-md-between{align-content:space-between!important}.align-content-md-around{align-content:space-around!important}.align-content-md-stretch{align-content:stretch!important}.align-self-md-auto{align-self:auto!important}.align-self-md-start{align-self:flex-start!important}.align-self-md-end{align-self:flex-end!important}.align-self-md-center{align-self:center!important}.align-self-md-baseline{align-self:baseline!important}.align-self-md-stretch{align-self:stretch!important}.order-md-first{order:-1!important}.order-md-0{order:0!important}.order-md-1{order:1!important}.order-md-2{order:2!important}.order-md-3{order:3!important}.order-md-4{order:4!important}.order-md-5{order:5!important}.order-md-last{order:6!important}.m-md-0{margin:0!important}.m-md-1{margin:.25rem!important}.m-md-2{margin:.5rem!important}.m-md-3{margin:1rem!important}.m-md-4{margin:1.5rem!important}.m-md-5{margin:3rem!important}.m-md-auto{margin:auto!important}.mx-md-0{margin-right:0!important;margin-left:0!important}.mx-md-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-md-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-md-3{margin-right:1rem!important;margin-left:1rem!important}.mx-md-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-md-5{margin-right:3rem!important;margin-left:3rem!important}.mx-md-auto{margin-right:auto!important;margin-left:auto!important}.my-md-0{margin-top:0!important;margin-bottom:0!important}.my-md-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-md-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-md-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-md-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-md-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-md-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-md-0{margin-top:0!important}.mt-md-1{margin-top:.25rem!important}.mt-md-2{margin-top:.5rem!important}.mt-md-3{margin-top:1rem!important}.mt-md-4{margin-top:1.5rem!important}.mt-md-5{margin-top:3rem!important}.mt-md-auto{margin-top:auto!important}.me-md-0{margin-right:0!important}.me-md-1{margin-right:.25rem!important}.me-md-2{margin-right:.5rem!important}.me-md-3{margin-right:1rem!important}.me-md-4{margin-right:1.5rem!important}.me-md-5{margin-right:3rem!important}.me-md-auto{margin-right:auto!important}.mb-md-0{margin-bottom:0!important}.mb-md-1{margin-bottom:.25rem!important}.mb-md-2{margin-bottom:.5rem!important}.mb-md-3{margin-bottom:1rem!important}.mb-md-4{margin-bottom:1.5rem!important}.mb-md-5{margin-bottom:3rem!important}.mb-md-auto{margin-bottom:auto!important}.ms-md-0{margin-left:0!important}.ms-md-1{margin-left:.25rem!important}.ms-md-2{margin-left:.5rem!important}.ms-md-3{margin-left:1rem!important}.ms-md-4{margin-left:1.5rem!important}.ms-md-5{margin-left:3rem!important}.ms-md-auto{margin-left:auto!important}.p-md-0{padding:0!important}.p-md-1{padding:.25rem!important}.p-md-2{padding:.5rem!important}.p-md-3{padding:1rem!important}.p-md-4{padding:1.5rem!important}.p-md-5{padding:3rem!important}.px-md-0{padding-right:0!important;padding-left:0!important}.px-md-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-md-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-md-3{padding-right:1rem!important;padding-left:1rem!important}.px-md-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-md-5{padding-right:3rem!important;padding-left:3rem!important}.py-md-0{padding-top:0!important;padding-bottom:0!important}.py-md-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-md-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-md-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-md-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-md-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-md-0{padding-top:0!important}.pt-md-1{padding-top:.25rem!important}.pt-md-2{padding-top:.5rem!important}.pt-md-3{padding-top:1rem!important}.pt-md-4{padding-top:1.5rem!important}.pt-md-5{padding-top:3rem!important}.pe-md-0{padding-right:0!important}.pe-md-1{padding-right:.25rem!important}.pe-md-2{padding-right:.5rem!important}.pe-md-3{padding-right:1rem!important}.pe-md-4{padding-right:1.5rem!important}.pe-md-5{padding-right:3rem!important}.pb-md-0{padding-bottom:0!important}.pb-md-1{padding-bottom:.25rem!important}.pb-md-2{padding-bottom:.5rem!important}.pb-md-3{padding-bottom:1rem!important}.pb-md-4{padding-bottom:1.5rem!important}.pb-md-5{padding-bottom:3rem!important}.ps-md-0{padding-left:0!important}.ps-md-1{padding-left:.25rem!important}.ps-md-2{padding-left:.5rem!important}.ps-md-3{padding-left:1rem!important}.ps-md-4{padding-left:1.5rem!important}.ps-md-5{padding-left:3rem!important}.gap-md-0{gap:0!important}.gap-md-1{gap:.25rem!important}.gap-md-2{gap:.5rem!important}.gap-md-3{gap:1rem!important}.gap-md-4{gap:1.5rem!important}.gap-md-5{gap:3rem!important}.row-gap-md-0{row-gap:0!important}.row-gap-md-1{row-gap:.25rem!important}.row-gap-md-2{row-gap:.5rem!important}.row-gap-md-3{row-gap:1rem!important}.row-gap-md-4{row-gap:1.5rem!important}.row-gap-md-5{row-gap:3rem!important}.column-gap-md-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-md-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-md-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-md-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-md-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-md-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.text-md-start{text-align:left!important}.text-md-end{text-align:right!important}.text-md-center{text-align:center!important}}@media (min-width:992px){.float-lg-start{float:left!important}.float-lg-end{float:right!important}.float-lg-none{float:none!important}.object-fit-lg-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-lg-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-lg-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-lg-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-lg-none{-o-object-fit:none!important;object-fit:none!important}.d-lg-inline{display:inline!important}.d-lg-inline-block{display:inline-block!important}.d-lg-block{display:block!important}.d-lg-grid{display:grid!important}.d-lg-inline-grid{display:inline-grid!important}.d-lg-table{display:table!important}.d-lg-table-row{display:table-row!important}.d-lg-table-cell{display:table-cell!important}.d-lg-flex{display:flex!important}.d-lg-inline-flex{display:inline-flex!important}.d-lg-none{display:none!important}.flex-lg-fill{flex:1 1 auto!important}.flex-lg-row{flex-direction:row!important}.flex-lg-column{flex-direction:column!important}.flex-lg-row-reverse{flex-direction:row-reverse!important}.flex-lg-column-reverse{flex-direction:column-reverse!important}.flex-lg-grow-0{flex-grow:0!important}.flex-lg-grow-1{flex-grow:1!important}.flex-lg-shrink-0{flex-shrink:0!important}.flex-lg-shrink-1{flex-shrink:1!important}.flex-lg-wrap{flex-wrap:wrap!important}.flex-lg-nowrap{flex-wrap:nowrap!important}.flex-lg-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-lg-start{justify-content:flex-start!important}.justify-content-lg-end{justify-content:flex-end!important}.justify-content-lg-center{justify-content:center!important}.justify-content-lg-between{justify-content:space-between!important}.justify-content-lg-around{justify-content:space-around!important}.justify-content-lg-evenly{justify-content:space-evenly!important}.align-items-lg-start{align-items:flex-start!important}.align-items-lg-end{align-items:flex-end!important}.align-items-lg-center{align-items:center!important}.align-items-lg-baseline{align-items:baseline!important}.align-items-lg-stretch{align-items:stretch!important}.align-content-lg-start{align-content:flex-start!important}.align-content-lg-end{align-content:flex-end!important}.align-content-lg-center{align-content:center!important}.align-content-lg-between{align-content:space-between!important}.align-content-lg-around{align-content:space-around!important}.align-content-lg-stretch{align-content:stretch!important}.align-self-lg-auto{align-self:auto!important}.align-self-lg-start{align-self:flex-start!important}.align-self-lg-end{align-self:flex-end!important}.align-self-lg-center{align-self:center!important}.align-self-lg-baseline{align-self:baseline!important}.align-self-lg-stretch{align-self:stretch!important}.order-lg-first{order:-1!important}.order-lg-0{order:0!important}.order-lg-1{order:1!important}.order-lg-2{order:2!important}.order-lg-3{order:3!important}.order-lg-4{order:4!important}.order-lg-5{order:5!important}.order-lg-last{order:6!important}.m-lg-0{margin:0!important}.m-lg-1{margin:.25rem!important}.m-lg-2{margin:.5rem!important}.m-lg-3{margin:1rem!important}.m-lg-4{margin:1.5rem!important}.m-lg-5{margin:3rem!important}.m-lg-auto{margin:auto!important}.mx-lg-0{margin-right:0!important;margin-left:0!important}.mx-lg-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-lg-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-lg-3{margin-right:1rem!important;margin-left:1rem!important}.mx-lg-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-lg-5{margin-right:3rem!important;margin-left:3rem!important}.mx-lg-auto{margin-right:auto!important;margin-left:auto!important}.my-lg-0{margin-top:0!important;margin-bottom:0!important}.my-lg-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-lg-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-lg-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-lg-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-lg-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-lg-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-lg-0{margin-top:0!important}.mt-lg-1{margin-top:.25rem!important}.mt-lg-2{margin-top:.5rem!important}.mt-lg-3{margin-top:1rem!important}.mt-lg-4{margin-top:1.5rem!important}.mt-lg-5{margin-top:3rem!important}.mt-lg-auto{margin-top:auto!important}.me-lg-0{margin-right:0!important}.me-lg-1{margin-right:.25rem!important}.me-lg-2{margin-right:.5rem!important}.me-lg-3{margin-right:1rem!important}.me-lg-4{margin-right:1.5rem!important}.me-lg-5{margin-right:3rem!important}.me-lg-auto{margin-right:auto!important}.mb-lg-0{margin-bottom:0!important}.mb-lg-1{margin-bottom:.25rem!important}.mb-lg-2{margin-bottom:.5rem!important}.mb-lg-3{margin-bottom:1rem!important}.mb-lg-4{margin-bottom:1.5rem!important}.mb-lg-5{margin-bottom:3rem!important}.mb-lg-auto{margin-bottom:auto!important}.ms-lg-0{margin-left:0!important}.ms-lg-1{margin-left:.25rem!important}.ms-lg-2{margin-left:.5rem!important}.ms-lg-3{margin-left:1rem!important}.ms-lg-4{margin-left:1.5rem!important}.ms-lg-5{margin-left:3rem!important}.ms-lg-auto{margin-left:auto!important}.p-lg-0{padding:0!important}.p-lg-1{padding:.25rem!important}.p-lg-2{padding:.5rem!important}.p-lg-3{padding:1rem!important}.p-lg-4{padding:1.5rem!important}.p-lg-5{padding:3rem!important}.px-lg-0{padding-right:0!important;padding-left:0!important}.px-lg-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-lg-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-lg-3{padding-right:1rem!important;padding-left:1rem!important}.px-lg-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-lg-5{padding-right:3rem!important;padding-left:3rem!important}.py-lg-0{padding-top:0!important;padding-bottom:0!important}.py-lg-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-lg-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-lg-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-lg-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-lg-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-lg-0{padding-top:0!important}.pt-lg-1{padding-top:.25rem!important}.pt-lg-2{padding-top:.5rem!important}.pt-lg-3{padding-top:1rem!important}.pt-lg-4{padding-top:1.5rem!important}.pt-lg-5{padding-top:3rem!important}.pe-lg-0{padding-right:0!important}.pe-lg-1{padding-right:.25rem!important}.pe-lg-2{padding-right:.5rem!important}.pe-lg-3{padding-right:1rem!important}.pe-lg-4{padding-right:1.5rem!important}.pe-lg-5{padding-right:3rem!important}.pb-lg-0{padding-bottom:0!important}.pb-lg-1{padding-bottom:.25rem!important}.pb-lg-2{padding-bottom:.5rem!important}.pb-lg-3{padding-bottom:1rem!important}.pb-lg-4{padding-bottom:1.5rem!important}.pb-lg-5{padding-bottom:3rem!important}.ps-lg-0{padding-left:0!important}.ps-lg-1{padding-left:.25rem!important}.ps-lg-2{padding-left:.5rem!important}.ps-lg-3{padding-left:1rem!important}.ps-lg-4{padding-left:1.5rem!important}.ps-lg-5{padding-left:3rem!important}.gap-lg-0{gap:0!important}.gap-lg-1{gap:.25rem!important}.gap-lg-2{gap:.5rem!important}.gap-lg-3{gap:1rem!important}.gap-lg-4{gap:1.5rem!important}.gap-lg-5{gap:3rem!important}.row-gap-lg-0{row-gap:0!important}.row-gap-lg-1{row-gap:.25rem!important}.row-gap-lg-2{row-gap:.5rem!important}.row-gap-lg-3{row-gap:1rem!important}.row-gap-lg-4{row-gap:1.5rem!important}.row-gap-lg-5{row-gap:3rem!important}.column-gap-lg-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-lg-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-lg-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-lg-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-lg-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-lg-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.text-lg-start{text-align:left!important}.text-lg-end{text-align:right!important}.text-lg-center{text-align:center!important}}@media (min-width:1200px){.float-xl-start{float:left!important}.float-xl-end{float:right!important}.float-xl-none{float:none!important}.object-fit-xl-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-xl-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-xl-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-xl-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-xl-none{-o-object-fit:none!important;object-fit:none!important}.d-xl-inline{display:inline!important}.d-xl-inline-block{display:inline-block!important}.d-xl-block{display:block!important}.d-xl-grid{display:grid!important}.d-xl-inline-grid{display:inline-grid!important}.d-xl-table{display:table!important}.d-xl-table-row{display:table-row!important}.d-xl-table-cell{display:table-cell!important}.d-xl-flex{display:flex!important}.d-xl-inline-flex{display:inline-flex!important}.d-xl-none{display:none!important}.flex-xl-fill{flex:1 1 auto!important}.flex-xl-row{flex-direction:row!important}.flex-xl-column{flex-direction:column!important}.flex-xl-row-reverse{flex-direction:row-reverse!important}.flex-xl-column-reverse{flex-direction:column-reverse!important}.flex-xl-grow-0{flex-grow:0!important}.flex-xl-grow-1{flex-grow:1!important}.flex-xl-shrink-0{flex-shrink:0!important}.flex-xl-shrink-1{flex-shrink:1!important}.flex-xl-wrap{flex-wrap:wrap!important}.flex-xl-nowrap{flex-wrap:nowrap!important}.flex-xl-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-xl-start{justify-content:flex-start!important}.justify-content-xl-end{justify-content:flex-end!important}.justify-content-xl-center{justify-content:center!important}.justify-content-xl-between{justify-content:space-between!important}.justify-content-xl-around{justify-content:space-around!important}.justify-content-xl-evenly{justify-content:space-evenly!important}.align-items-xl-start{align-items:flex-start!important}.align-items-xl-end{align-items:flex-end!important}.align-items-xl-center{align-items:center!important}.align-items-xl-baseline{align-items:baseline!important}.align-items-xl-stretch{align-items:stretch!important}.align-content-xl-start{align-content:flex-start!important}.align-content-xl-end{align-content:flex-end!important}.align-content-xl-center{align-content:center!important}.align-content-xl-between{align-content:space-between!important}.align-content-xl-around{align-content:space-around!important}.align-content-xl-stretch{align-content:stretch!important}.align-self-xl-auto{align-self:auto!important}.align-self-xl-start{align-self:flex-start!important}.align-self-xl-end{align-self:flex-end!important}.align-self-xl-center{align-self:center!important}.align-self-xl-baseline{align-self:baseline!important}.align-self-xl-stretch{align-self:stretch!important}.order-xl-first{order:-1!important}.order-xl-0{order:0!important}.order-xl-1{order:1!important}.order-xl-2{order:2!important}.order-xl-3{order:3!important}.order-xl-4{order:4!important}.order-xl-5{order:5!important}.order-xl-last{order:6!important}.m-xl-0{margin:0!important}.m-xl-1{margin:.25rem!important}.m-xl-2{margin:.5rem!important}.m-xl-3{margin:1rem!important}.m-xl-4{margin:1.5rem!important}.m-xl-5{margin:3rem!important}.m-xl-auto{margin:auto!important}.mx-xl-0{margin-right:0!important;margin-left:0!important}.mx-xl-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-xl-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-xl-3{margin-right:1rem!important;margin-left:1rem!important}.mx-xl-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-xl-5{margin-right:3rem!important;margin-left:3rem!important}.mx-xl-auto{margin-right:auto!important;margin-left:auto!important}.my-xl-0{margin-top:0!important;margin-bottom:0!important}.my-xl-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-xl-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-xl-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-xl-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-xl-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-xl-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-xl-0{margin-top:0!important}.mt-xl-1{margin-top:.25rem!important}.mt-xl-2{margin-top:.5rem!important}.mt-xl-3{margin-top:1rem!important}.mt-xl-4{margin-top:1.5rem!important}.mt-xl-5{margin-top:3rem!important}.mt-xl-auto{margin-top:auto!important}.me-xl-0{margin-right:0!important}.me-xl-1{margin-right:.25rem!important}.me-xl-2{margin-right:.5rem!important}.me-xl-3{margin-right:1rem!important}.me-xl-4{margin-right:1.5rem!important}.me-xl-5{margin-right:3rem!important}.me-xl-auto{margin-right:auto!important}.mb-xl-0{margin-bottom:0!important}.mb-xl-1{margin-bottom:.25rem!important}.mb-xl-2{margin-bottom:.5rem!important}.mb-xl-3{margin-bottom:1rem!important}.mb-xl-4{margin-bottom:1.5rem!important}.mb-xl-5{margin-bottom:3rem!important}.mb-xl-auto{margin-bottom:auto!important}.ms-xl-0{margin-left:0!important}.ms-xl-1{margin-left:.25rem!important}.ms-xl-2{margin-left:.5rem!important}.ms-xl-3{margin-left:1rem!important}.ms-xl-4{margin-left:1.5rem!important}.ms-xl-5{margin-left:3rem!important}.ms-xl-auto{margin-left:auto!important}.p-xl-0{padding:0!important}.p-xl-1{padding:.25rem!important}.p-xl-2{padding:.5rem!important}.p-xl-3{padding:1rem!important}.p-xl-4{padding:1.5rem!important}.p-xl-5{padding:3rem!important}.px-xl-0{padding-right:0!important;padding-left:0!important}.px-xl-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-xl-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-xl-3{padding-right:1rem!important;padding-left:1rem!important}.px-xl-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-xl-5{padding-right:3rem!important;padding-left:3rem!important}.py-xl-0{padding-top:0!important;padding-bottom:0!important}.py-xl-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-xl-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-xl-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-xl-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-xl-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-xl-0{padding-top:0!important}.pt-xl-1{padding-top:.25rem!important}.pt-xl-2{padding-top:.5rem!important}.pt-xl-3{padding-top:1rem!important}.pt-xl-4{padding-top:1.5rem!important}.pt-xl-5{padding-top:3rem!important}.pe-xl-0{padding-right:0!important}.pe-xl-1{padding-right:.25rem!important}.pe-xl-2{padding-right:.5rem!important}.pe-xl-3{padding-right:1rem!important}.pe-xl-4{padding-right:1.5rem!important}.pe-xl-5{padding-right:3rem!important}.pb-xl-0{padding-bottom:0!important}.pb-xl-1{padding-bottom:.25rem!important}.pb-xl-2{padding-bottom:.5rem!important}.pb-xl-3{padding-bottom:1rem!important}.pb-xl-4{padding-bottom:1.5rem!important}.pb-xl-5{padding-bottom:3rem!important}.ps-xl-0{padding-left:0!important}.ps-xl-1{padding-left:.25rem!important}.ps-xl-2{padding-left:.5rem!important}.ps-xl-3{padding-left:1rem!important}.ps-xl-4{padding-left:1.5rem!important}.ps-xl-5{padding-left:3rem!important}.gap-xl-0{gap:0!important}.gap-xl-1{gap:.25rem!important}.gap-xl-2{gap:.5rem!important}.gap-xl-3{gap:1rem!important}.gap-xl-4{gap:1.5rem!important}.gap-xl-5{gap:3rem!important}.row-gap-xl-0{row-gap:0!important}.row-gap-xl-1{row-gap:.25rem!important}.row-gap-xl-2{row-gap:.5rem!important}.row-gap-xl-3{row-gap:1rem!important}.row-gap-xl-4{row-gap:1.5rem!important}.row-gap-xl-5{row-gap:3rem!important}.column-gap-xl-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-xl-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-xl-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-xl-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-xl-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-xl-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.text-xl-start{text-align:left!important}.text-xl-end{text-align:right!important}.text-xl-center{text-align:center!important}}@media (min-width:1400px){.float-xxl-start{float:left!important}.float-xxl-end{float:right!important}.float-xxl-none{float:none!important}.object-fit-xxl-contain{-o-object-fit:contain!important;object-fit:contain!important}.object-fit-xxl-cover{-o-object-fit:cover!important;object-fit:cover!important}.object-fit-xxl-fill{-o-object-fit:fill!important;object-fit:fill!important}.object-fit-xxl-scale{-o-object-fit:scale-down!important;object-fit:scale-down!important}.object-fit-xxl-none{-o-object-fit:none!important;object-fit:none!important}.d-xxl-inline{display:inline!important}.d-xxl-inline-block{display:inline-block!important}.d-xxl-block{display:block!important}.d-xxl-grid{display:grid!important}.d-xxl-inline-grid{display:inline-grid!important}.d-xxl-table{display:table!important}.d-xxl-table-row{display:table-row!important}.d-xxl-table-cell{display:table-cell!important}.d-xxl-flex{display:flex!important}.d-xxl-inline-flex{display:inline-flex!important}.d-xxl-none{display:none!important}.flex-xxl-fill{flex:1 1 auto!important}.flex-xxl-row{flex-direction:row!important}.flex-xxl-column{flex-direction:column!important}.flex-xxl-row-reverse{flex-direction:row-reverse!important}.flex-xxl-column-reverse{flex-direction:column-reverse!important}.flex-xxl-grow-0{flex-grow:0!important}.flex-xxl-grow-1{flex-grow:1!important}.flex-xxl-shrink-0{flex-shrink:0!important}.flex-xxl-shrink-1{flex-shrink:1!important}.flex-xxl-wrap{flex-wrap:wrap!important}.flex-xxl-nowrap{flex-wrap:nowrap!important}.flex-xxl-wrap-reverse{flex-wrap:wrap-reverse!important}.justify-content-xxl-start{justify-content:flex-start!important}.justify-content-xxl-end{justify-content:flex-end!important}.justify-content-xxl-center{justify-content:center!important}.justify-content-xxl-between{justify-content:space-between!important}.justify-content-xxl-around{justify-content:space-around!important}.justify-content-xxl-evenly{justify-content:space-evenly!important}.align-items-xxl-start{align-items:flex-start!important}.align-items-xxl-end{align-items:flex-end!important}.align-items-xxl-center{align-items:center!important}.align-items-xxl-baseline{align-items:baseline!important}.align-items-xxl-stretch{align-items:stretch!important}.align-content-xxl-start{align-content:flex-start!important}.align-content-xxl-end{align-content:flex-end!important}.align-content-xxl-center{align-content:center!important}.align-content-xxl-between{align-content:space-between!important}.align-content-xxl-around{align-content:space-around!important}.align-content-xxl-stretch{align-content:stretch!important}.align-self-xxl-auto{align-self:auto!important}.align-self-xxl-start{align-self:flex-start!important}.align-self-xxl-end{align-self:flex-end!important}.align-self-xxl-center{align-self:center!important}.align-self-xxl-baseline{align-self:baseline!important}.align-self-xxl-stretch{align-self:stretch!important}.order-xxl-first{order:-1!important}.order-xxl-0{order:0!important}.order-xxl-1{order:1!important}.order-xxl-2{order:2!important}.order-xxl-3{order:3!important}.order-xxl-4{order:4!important}.order-xxl-5{order:5!important}.order-xxl-last{order:6!important}.m-xxl-0{margin:0!important}.m-xxl-1{margin:.25rem!important}.m-xxl-2{margin:.5rem!important}.m-xxl-3{margin:1rem!important}.m-xxl-4{margin:1.5rem!important}.m-xxl-5{margin:3rem!important}.m-xxl-auto{margin:auto!important}.mx-xxl-0{margin-right:0!important;margin-left:0!important}.mx-xxl-1{margin-right:.25rem!important;margin-left:.25rem!important}.mx-xxl-2{margin-right:.5rem!important;margin-left:.5rem!important}.mx-xxl-3{margin-right:1rem!important;margin-left:1rem!important}.mx-xxl-4{margin-right:1.5rem!important;margin-left:1.5rem!important}.mx-xxl-5{margin-right:3rem!important;margin-left:3rem!important}.mx-xxl-auto{margin-right:auto!important;margin-left:auto!important}.my-xxl-0{margin-top:0!important;margin-bottom:0!important}.my-xxl-1{margin-top:.25rem!important;margin-bottom:.25rem!important}.my-xxl-2{margin-top:.5rem!important;margin-bottom:.5rem!important}.my-xxl-3{margin-top:1rem!important;margin-bottom:1rem!important}.my-xxl-4{margin-top:1.5rem!important;margin-bottom:1.5rem!important}.my-xxl-5{margin-top:3rem!important;margin-bottom:3rem!important}.my-xxl-auto{margin-top:auto!important;margin-bottom:auto!important}.mt-xxl-0{margin-top:0!important}.mt-xxl-1{margin-top:.25rem!important}.mt-xxl-2{margin-top:.5rem!important}.mt-xxl-3{margin-top:1rem!important}.mt-xxl-4{margin-top:1.5rem!important}.mt-xxl-5{margin-top:3rem!important}.mt-xxl-auto{margin-top:auto!important}.me-xxl-0{margin-right:0!important}.me-xxl-1{margin-right:.25rem!important}.me-xxl-2{margin-right:.5rem!important}.me-xxl-3{margin-right:1rem!important}.me-xxl-4{margin-right:1.5rem!important}.me-xxl-5{margin-right:3rem!important}.me-xxl-auto{margin-right:auto!important}.mb-xxl-0{margin-bottom:0!important}.mb-xxl-1{margin-bottom:.25rem!important}.mb-xxl-2{margin-bottom:.5rem!important}.mb-xxl-3{margin-bottom:1rem!important}.mb-xxl-4{margin-bottom:1.5rem!important}.mb-xxl-5{margin-bottom:3rem!important}.mb-xxl-auto{margin-bottom:auto!important}.ms-xxl-0{margin-left:0!important}.ms-xxl-1{margin-left:.25rem!important}.ms-xxl-2{margin-left:.5rem!important}.ms-xxl-3{margin-left:1rem!important}.ms-xxl-4{margin-left:1.5rem!important}.ms-xxl-5{margin-left:3rem!important}.ms-xxl-auto{margin-left:auto!important}.p-xxl-0{padding:0!important}.p-xxl-1{padding:.25rem!important}.p-xxl-2{padding:.5rem!important}.p-xxl-3{padding:1rem!important}.p-xxl-4{padding:1.5rem!important}.p-xxl-5{padding:3rem!important}.px-xxl-0{padding-right:0!important;padding-left:0!important}.px-xxl-1{padding-right:.25rem!important;padding-left:.25rem!important}.px-xxl-2{padding-right:.5rem!important;padding-left:.5rem!important}.px-xxl-3{padding-right:1rem!important;padding-left:1rem!important}.px-xxl-4{padding-right:1.5rem!important;padding-left:1.5rem!important}.px-xxl-5{padding-right:3rem!important;padding-left:3rem!important}.py-xxl-0{padding-top:0!important;padding-bottom:0!important}.py-xxl-1{padding-top:.25rem!important;padding-bottom:.25rem!important}.py-xxl-2{padding-top:.5rem!important;padding-bottom:.5rem!important}.py-xxl-3{padding-top:1rem!important;padding-bottom:1rem!important}.py-xxl-4{padding-top:1.5rem!important;padding-bottom:1.5rem!important}.py-xxl-5{padding-top:3rem!important;padding-bottom:3rem!important}.pt-xxl-0{padding-top:0!important}.pt-xxl-1{padding-top:.25rem!important}.pt-xxl-2{padding-top:.5rem!important}.pt-xxl-3{padding-top:1rem!important}.pt-xxl-4{padding-top:1.5rem!important}.pt-xxl-5{padding-top:3rem!important}.pe-xxl-0{padding-right:0!important}.pe-xxl-1{padding-right:.25rem!important}.pe-xxl-2{padding-right:.5rem!important}.pe-xxl-3{padding-right:1rem!important}.pe-xxl-4{padding-right:1.5rem!important}.pe-xxl-5{padding-right:3rem!important}.pb-xxl-0{padding-bottom:0!important}.pb-xxl-1{padding-bottom:.25rem!important}.pb-xxl-2{padding-bottom:.5rem!important}.pb-xxl-3{padding-bottom:1rem!important}.pb-xxl-4{padding-bottom:1.5rem!important}.pb-xxl-5{padding-bottom:3rem!important}.ps-xxl-0{padding-left:0!important}.ps-xxl-1{padding-left:.25rem!important}.ps-xxl-2{padding-left:.5rem!important}.ps-xxl-3{padding-left:1rem!important}.ps-xxl-4{padding-left:1.5rem!important}.ps-xxl-5{padding-left:3rem!important}.gap-xxl-0{gap:0!important}.gap-xxl-1{gap:.25rem!important}.gap-xxl-2{gap:.5rem!important}.gap-xxl-3{gap:1rem!important}.gap-xxl-4{gap:1.5rem!important}.gap-xxl-5{gap:3rem!important}.row-gap-xxl-0{row-gap:0!important}.row-gap-xxl-1{row-gap:.25rem!important}.row-gap-xxl-2{row-gap:.5rem!important}.row-gap-xxl-3{row-gap:1rem!important}.row-gap-xxl-4{row-gap:1.5rem!important}.row-gap-xxl-5{row-gap:3rem!important}.column-gap-xxl-0{-moz-column-gap:0!important;column-gap:0!important}.column-gap-xxl-1{-moz-column-gap:0.25rem!important;column-gap:.25rem!important}.column-gap-xxl-2{-moz-column-gap:0.5rem!important;column-gap:.5rem!important}.column-gap-xxl-3{-moz-column-gap:1rem!important;column-gap:1rem!important}.column-gap-xxl-4{-moz-column-gap:1.5rem!important;column-gap:1.5rem!important}.column-gap-xxl-5{-moz-column-gap:3rem!important;column-gap:3rem!important}.text-xxl-start{text-align:left!important}.text-xxl-end{text-align:right!important}.text-xxl-center{text-align:center!important}}@media (min-width:1200px){.fs-1{font-size:2.5rem!important}.fs-2{font-size:2rem!important}.fs-3{font-size:1.75rem!important}.fs-4{font-size:1.5rem!important}}@media print{.d-print-inline{display:inline!important}.d-print-inline-block{display:inline-block!important}.d-print-block{display:block!important}.d-print-grid{display:grid!important}.d-print-inline-grid{display:inline-grid!important}.d-print-table{display:table!important}.d-print-table-row{display:table-row!important}.d-print-table-cell{display:table-cell!important}.d-print-flex{display:flex!important}.d-print-inline-flex{display:inline-flex!important}.d-print-none{display:none!important}} +/*# sourceMappingURL=bootstrap.min.css.map */ \ No newline at end of file diff --git a/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/js/bootstrap.bundle.min.js b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/js/bootstrap.bundle.min.js new file mode 100644 index 0000000..04e9185 --- /dev/null +++ b/ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/wwwroot/lib/bootstrap/js/bootstrap.bundle.min.js @@ -0,0 +1,7 @@ +/*! + * Bootstrap v5.3.3 (https://getbootstrap.com/) + * Copyright 2011-2024 The Bootstrap Authors (https://github.com/twbs/bootstrap/graphs/contributors) + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE) + */ +!function(t,e){"object"==typeof exports&&"undefined"!=typeof module?module.exports=e():"function"==typeof define&&define.amd?define(e):(t="undefined"!=typeof globalThis?globalThis:t||self).bootstrap=e()}(this,(function(){"use strict";const t=new Map,e={set(e,i,n){t.has(e)||t.set(e,new Map);const s=t.get(e);s.has(i)||0===s.size?s.set(i,n):console.error(`Bootstrap doesn't allow more than one instance per element. Bound instance: ${Array.from(s.keys())[0]}.`)},get:(e,i)=>t.has(e)&&t.get(e).get(i)||null,remove(e,i){if(!t.has(e))return;const n=t.get(e);n.delete(i),0===n.size&&t.delete(e)}},i="transitionend",n=t=>(t&&window.CSS&&window.CSS.escape&&(t=t.replace(/#([^\s"#']+)/g,((t,e)=>`#${CSS.escape(e)}`))),t),s=t=>{t.dispatchEvent(new Event(i))},o=t=>!(!t||"object"!=typeof t)&&(void 0!==t.jquery&&(t=t[0]),void 0!==t.nodeType),r=t=>o(t)?t.jquery?t[0]:t:"string"==typeof t&&t.length>0?document.querySelector(n(t)):null,a=t=>{if(!o(t)||0===t.getClientRects().length)return!1;const e="visible"===getComputedStyle(t).getPropertyValue("visibility"),i=t.closest("details:not([open])");if(!i)return e;if(i!==t){const e=t.closest("summary");if(e&&e.parentNode!==i)return!1;if(null===e)return!1}return e},l=t=>!t||t.nodeType!==Node.ELEMENT_NODE||!!t.classList.contains("disabled")||(void 0!==t.disabled?t.disabled:t.hasAttribute("disabled")&&"false"!==t.getAttribute("disabled")),c=t=>{if(!document.documentElement.attachShadow)return null;if("function"==typeof t.getRootNode){const e=t.getRootNode();return e instanceof ShadowRoot?e:null}return t instanceof ShadowRoot?t:t.parentNode?c(t.parentNode):null},h=()=>{},d=t=>{t.offsetHeight},u=()=>window.jQuery&&!document.body.hasAttribute("data-bs-no-jquery")?window.jQuery:null,f=[],p=()=>"rtl"===document.documentElement.dir,m=t=>{var e;e=()=>{const e=u();if(e){const i=t.NAME,n=e.fn[i];e.fn[i]=t.jQueryInterface,e.fn[i].Constructor=t,e.fn[i].noConflict=()=>(e.fn[i]=n,t.jQueryInterface)}},"loading"===document.readyState?(f.length||document.addEventListener("DOMContentLoaded",(()=>{for(const t of f)t()})),f.push(e)):e()},g=(t,e=[],i=t)=>"function"==typeof t?t(...e):i,_=(t,e,n=!0)=>{if(!n)return void g(t);const o=(t=>{if(!t)return 0;let{transitionDuration:e,transitionDelay:i}=window.getComputedStyle(t);const n=Number.parseFloat(e),s=Number.parseFloat(i);return n||s?(e=e.split(",")[0],i=i.split(",")[0],1e3*(Number.parseFloat(e)+Number.parseFloat(i))):0})(e)+5;let r=!1;const a=({target:n})=>{n===e&&(r=!0,e.removeEventListener(i,a),g(t))};e.addEventListener(i,a),setTimeout((()=>{r||s(e)}),o)},b=(t,e,i,n)=>{const s=t.length;let o=t.indexOf(e);return-1===o?!i&&n?t[s-1]:t[0]:(o+=i?1:-1,n&&(o=(o+s)%s),t[Math.max(0,Math.min(o,s-1))])},v=/[^.]*(?=\..*)\.|.*/,y=/\..*/,w=/::\d+$/,A={};let E=1;const T={mouseenter:"mouseover",mouseleave:"mouseout"},C=new Set(["click","dblclick","mouseup","mousedown","contextmenu","mousewheel","DOMMouseScroll","mouseover","mouseout","mousemove","selectstart","selectend","keydown","keypress","keyup","orientationchange","touchstart","touchmove","touchend","touchcancel","pointerdown","pointermove","pointerup","pointerleave","pointercancel","gesturestart","gesturechange","gestureend","focus","blur","change","reset","select","submit","focusin","focusout","load","unload","beforeunload","resize","move","DOMContentLoaded","readystatechange","error","abort","scroll"]);function O(t,e){return e&&`${e}::${E++}`||t.uidEvent||E++}function x(t){const e=O(t);return t.uidEvent=e,A[e]=A[e]||{},A[e]}function k(t,e,i=null){return Object.values(t).find((t=>t.callable===e&&t.delegationSelector===i))}function L(t,e,i){const n="string"==typeof e,s=n?i:e||i;let o=I(t);return C.has(o)||(o=t),[n,s,o]}function S(t,e,i,n,s){if("string"!=typeof e||!t)return;let[o,r,a]=L(e,i,n);if(e in T){const t=t=>function(e){if(!e.relatedTarget||e.relatedTarget!==e.delegateTarget&&!e.delegateTarget.contains(e.relatedTarget))return t.call(this,e)};r=t(r)}const l=x(t),c=l[a]||(l[a]={}),h=k(c,r,o?i:null);if(h)return void(h.oneOff=h.oneOff&&s);const d=O(r,e.replace(v,"")),u=o?function(t,e,i){return function n(s){const o=t.querySelectorAll(e);for(let{target:r}=s;r&&r!==this;r=r.parentNode)for(const a of o)if(a===r)return P(s,{delegateTarget:r}),n.oneOff&&N.off(t,s.type,e,i),i.apply(r,[s])}}(t,i,r):function(t,e){return function i(n){return P(n,{delegateTarget:t}),i.oneOff&&N.off(t,n.type,e),e.apply(t,[n])}}(t,r);u.delegationSelector=o?i:null,u.callable=r,u.oneOff=s,u.uidEvent=d,c[d]=u,t.addEventListener(a,u,o)}function D(t,e,i,n,s){const o=k(e[i],n,s);o&&(t.removeEventListener(i,o,Boolean(s)),delete e[i][o.uidEvent])}function $(t,e,i,n){const s=e[i]||{};for(const[o,r]of Object.entries(s))o.includes(n)&&D(t,e,i,r.callable,r.delegationSelector)}function I(t){return t=t.replace(y,""),T[t]||t}const N={on(t,e,i,n){S(t,e,i,n,!1)},one(t,e,i,n){S(t,e,i,n,!0)},off(t,e,i,n){if("string"!=typeof e||!t)return;const[s,o,r]=L(e,i,n),a=r!==e,l=x(t),c=l[r]||{},h=e.startsWith(".");if(void 0===o){if(h)for(const i of Object.keys(l))$(t,l,i,e.slice(1));for(const[i,n]of Object.entries(c)){const s=i.replace(w,"");a&&!e.includes(s)||D(t,l,r,n.callable,n.delegationSelector)}}else{if(!Object.keys(c).length)return;D(t,l,r,o,s?i:null)}},trigger(t,e,i){if("string"!=typeof e||!t)return null;const n=u();let s=null,o=!0,r=!0,a=!1;e!==I(e)&&n&&(s=n.Event(e,i),n(t).trigger(s),o=!s.isPropagationStopped(),r=!s.isImmediatePropagationStopped(),a=s.isDefaultPrevented());const l=P(new Event(e,{bubbles:o,cancelable:!0}),i);return a&&l.preventDefault(),r&&t.dispatchEvent(l),l.defaultPrevented&&s&&s.preventDefault(),l}};function P(t,e={}){for(const[i,n]of Object.entries(e))try{t[i]=n}catch(e){Object.defineProperty(t,i,{configurable:!0,get:()=>n})}return t}function j(t){if("true"===t)return!0;if("false"===t)return!1;if(t===Number(t).toString())return Number(t);if(""===t||"null"===t)return null;if("string"!=typeof t)return t;try{return JSON.parse(decodeURIComponent(t))}catch(e){return t}}function M(t){return t.replace(/[A-Z]/g,(t=>`-${t.toLowerCase()}`))}const F={setDataAttribute(t,e,i){t.setAttribute(`data-bs-${M(e)}`,i)},removeDataAttribute(t,e){t.removeAttribute(`data-bs-${M(e)}`)},getDataAttributes(t){if(!t)return{};const e={},i=Object.keys(t.dataset).filter((t=>t.startsWith("bs")&&!t.startsWith("bsConfig")));for(const n of i){let i=n.replace(/^bs/,"");i=i.charAt(0).toLowerCase()+i.slice(1,i.length),e[i]=j(t.dataset[n])}return e},getDataAttribute:(t,e)=>j(t.getAttribute(`data-bs-${M(e)}`))};class H{static get Default(){return{}}static get DefaultType(){return{}}static get NAME(){throw new Error('You have to implement the static method "NAME", for each component!')}_getConfig(t){return t=this._mergeConfigObj(t),t=this._configAfterMerge(t),this._typeCheckConfig(t),t}_configAfterMerge(t){return t}_mergeConfigObj(t,e){const i=o(e)?F.getDataAttribute(e,"config"):{};return{...this.constructor.Default,..."object"==typeof i?i:{},...o(e)?F.getDataAttributes(e):{},..."object"==typeof t?t:{}}}_typeCheckConfig(t,e=this.constructor.DefaultType){for(const[n,s]of Object.entries(e)){const e=t[n],r=o(e)?"element":null==(i=e)?`${i}`:Object.prototype.toString.call(i).match(/\s([a-z]+)/i)[1].toLowerCase();if(!new RegExp(s).test(r))throw new TypeError(`${this.constructor.NAME.toUpperCase()}: Option "${n}" provided type "${r}" but expected type "${s}".`)}var i}}class W extends H{constructor(t,i){super(),(t=r(t))&&(this._element=t,this._config=this._getConfig(i),e.set(this._element,this.constructor.DATA_KEY,this))}dispose(){e.remove(this._element,this.constructor.DATA_KEY),N.off(this._element,this.constructor.EVENT_KEY);for(const t of Object.getOwnPropertyNames(this))this[t]=null}_queueCallback(t,e,i=!0){_(t,e,i)}_getConfig(t){return t=this._mergeConfigObj(t,this._element),t=this._configAfterMerge(t),this._typeCheckConfig(t),t}static getInstance(t){return e.get(r(t),this.DATA_KEY)}static getOrCreateInstance(t,e={}){return this.getInstance(t)||new this(t,"object"==typeof e?e:null)}static get VERSION(){return"5.3.3"}static get DATA_KEY(){return`bs.${this.NAME}`}static get EVENT_KEY(){return`.${this.DATA_KEY}`}static eventName(t){return`${t}${this.EVENT_KEY}`}}const B=t=>{let e=t.getAttribute("data-bs-target");if(!e||"#"===e){let i=t.getAttribute("href");if(!i||!i.includes("#")&&!i.startsWith("."))return null;i.includes("#")&&!i.startsWith("#")&&(i=`#${i.split("#")[1]}`),e=i&&"#"!==i?i.trim():null}return e?e.split(",").map((t=>n(t))).join(","):null},z={find:(t,e=document.documentElement)=>[].concat(...Element.prototype.querySelectorAll.call(e,t)),findOne:(t,e=document.documentElement)=>Element.prototype.querySelector.call(e,t),children:(t,e)=>[].concat(...t.children).filter((t=>t.matches(e))),parents(t,e){const i=[];let n=t.parentNode.closest(e);for(;n;)i.push(n),n=n.parentNode.closest(e);return i},prev(t,e){let i=t.previousElementSibling;for(;i;){if(i.matches(e))return[i];i=i.previousElementSibling}return[]},next(t,e){let i=t.nextElementSibling;for(;i;){if(i.matches(e))return[i];i=i.nextElementSibling}return[]},focusableChildren(t){const e=["a","button","input","textarea","select","details","[tabindex]",'[contenteditable="true"]'].map((t=>`${t}:not([tabindex^="-"])`)).join(",");return this.find(e,t).filter((t=>!l(t)&&a(t)))},getSelectorFromElement(t){const e=B(t);return e&&z.findOne(e)?e:null},getElementFromSelector(t){const e=B(t);return e?z.findOne(e):null},getMultipleElementsFromSelector(t){const e=B(t);return e?z.find(e):[]}},R=(t,e="hide")=>{const i=`click.dismiss${t.EVENT_KEY}`,n=t.NAME;N.on(document,i,`[data-bs-dismiss="${n}"]`,(function(i){if(["A","AREA"].includes(this.tagName)&&i.preventDefault(),l(this))return;const s=z.getElementFromSelector(this)||this.closest(`.${n}`);t.getOrCreateInstance(s)[e]()}))},q=".bs.alert",V=`close${q}`,K=`closed${q}`;class Q extends W{static get NAME(){return"alert"}close(){if(N.trigger(this._element,V).defaultPrevented)return;this._element.classList.remove("show");const t=this._element.classList.contains("fade");this._queueCallback((()=>this._destroyElement()),this._element,t)}_destroyElement(){this._element.remove(),N.trigger(this._element,K),this.dispose()}static jQueryInterface(t){return this.each((function(){const e=Q.getOrCreateInstance(this);if("string"==typeof t){if(void 0===e[t]||t.startsWith("_")||"constructor"===t)throw new TypeError(`No method named "${t}"`);e[t](this)}}))}}R(Q,"close"),m(Q);const X='[data-bs-toggle="button"]';class Y extends W{static get NAME(){return"button"}toggle(){this._element.setAttribute("aria-pressed",this._element.classList.toggle("active"))}static jQueryInterface(t){return this.each((function(){const e=Y.getOrCreateInstance(this);"toggle"===t&&e[t]()}))}}N.on(document,"click.bs.button.data-api",X,(t=>{t.preventDefault();const e=t.target.closest(X);Y.getOrCreateInstance(e).toggle()})),m(Y);const U=".bs.swipe",G=`touchstart${U}`,J=`touchmove${U}`,Z=`touchend${U}`,tt=`pointerdown${U}`,et=`pointerup${U}`,it={endCallback:null,leftCallback:null,rightCallback:null},nt={endCallback:"(function|null)",leftCallback:"(function|null)",rightCallback:"(function|null)"};class st extends H{constructor(t,e){super(),this._element=t,t&&st.isSupported()&&(this._config=this._getConfig(e),this._deltaX=0,this._supportPointerEvents=Boolean(window.PointerEvent),this._initEvents())}static get Default(){return it}static get DefaultType(){return nt}static get NAME(){return"swipe"}dispose(){N.off(this._element,U)}_start(t){this._supportPointerEvents?this._eventIsPointerPenTouch(t)&&(this._deltaX=t.clientX):this._deltaX=t.touches[0].clientX}_end(t){this._eventIsPointerPenTouch(t)&&(this._deltaX=t.clientX-this._deltaX),this._handleSwipe(),g(this._config.endCallback)}_move(t){this._deltaX=t.touches&&t.touches.length>1?0:t.touches[0].clientX-this._deltaX}_handleSwipe(){const t=Math.abs(this._deltaX);if(t<=40)return;const e=t/this._deltaX;this._deltaX=0,e&&g(e>0?this._config.rightCallback:this._config.leftCallback)}_initEvents(){this._supportPointerEvents?(N.on(this._element,tt,(t=>this._start(t))),N.on(this._element,et,(t=>this._end(t))),this._element.classList.add("pointer-event")):(N.on(this._element,G,(t=>this._start(t))),N.on(this._element,J,(t=>this._move(t))),N.on(this._element,Z,(t=>this._end(t))))}_eventIsPointerPenTouch(t){return this._supportPointerEvents&&("pen"===t.pointerType||"touch"===t.pointerType)}static isSupported(){return"ontouchstart"in document.documentElement||navigator.maxTouchPoints>0}}const ot=".bs.carousel",rt=".data-api",at="next",lt="prev",ct="left",ht="right",dt=`slide${ot}`,ut=`slid${ot}`,ft=`keydown${ot}`,pt=`mouseenter${ot}`,mt=`mouseleave${ot}`,gt=`dragstart${ot}`,_t=`load${ot}${rt}`,bt=`click${ot}${rt}`,vt="carousel",yt="active",wt=".active",At=".carousel-item",Et=wt+At,Tt={ArrowLeft:ht,ArrowRight:ct},Ct={interval:5e3,keyboard:!0,pause:"hover",ride:!1,touch:!0,wrap:!0},Ot={interval:"(number|boolean)",keyboard:"boolean",pause:"(string|boolean)",ride:"(boolean|string)",touch:"boolean",wrap:"boolean"};class xt extends W{constructor(t,e){super(t,e),this._interval=null,this._activeElement=null,this._isSliding=!1,this.touchTimeout=null,this._swipeHelper=null,this._indicatorsElement=z.findOne(".carousel-indicators",this._element),this._addEventListeners(),this._config.ride===vt&&this.cycle()}static get Default(){return Ct}static get DefaultType(){return Ot}static get NAME(){return"carousel"}next(){this._slide(at)}nextWhenVisible(){!document.hidden&&a(this._element)&&this.next()}prev(){this._slide(lt)}pause(){this._isSliding&&s(this._element),this._clearInterval()}cycle(){this._clearInterval(),this._updateInterval(),this._interval=setInterval((()=>this.nextWhenVisible()),this._config.interval)}_maybeEnableCycle(){this._config.ride&&(this._isSliding?N.one(this._element,ut,(()=>this.cycle())):this.cycle())}to(t){const e=this._getItems();if(t>e.length-1||t<0)return;if(this._isSliding)return void N.one(this._element,ut,(()=>this.to(t)));const i=this._getItemIndex(this._getActive());if(i===t)return;const n=t>i?at:lt;this._slide(n,e[t])}dispose(){this._swipeHelper&&this._swipeHelper.dispose(),super.dispose()}_configAfterMerge(t){return t.defaultInterval=t.interval,t}_addEventListeners(){this._config.keyboard&&N.on(this._element,ft,(t=>this._keydown(t))),"hover"===this._config.pause&&(N.on(this._element,pt,(()=>this.pause())),N.on(this._element,mt,(()=>this._maybeEnableCycle()))),this._config.touch&&st.isSupported()&&this._addTouchEventListeners()}_addTouchEventListeners(){for(const t of z.find(".carousel-item img",this._element))N.on(t,gt,(t=>t.preventDefault()));const t={leftCallback:()=>this._slide(this._directionToOrder(ct)),rightCallback:()=>this._slide(this._directionToOrder(ht)),endCallback:()=>{"hover"===this._config.pause&&(this.pause(),this.touchTimeout&&clearTimeout(this.touchTimeout),this.touchTimeout=setTimeout((()=>this._maybeEnableCycle()),500+this._config.interval))}};this._swipeHelper=new st(this._element,t)}_keydown(t){if(/input|textarea/i.test(t.target.tagName))return;const e=Tt[t.key];e&&(t.preventDefault(),this._slide(this._directionToOrder(e)))}_getItemIndex(t){return this._getItems().indexOf(t)}_setActiveIndicatorElement(t){if(!this._indicatorsElement)return;const e=z.findOne(wt,this._indicatorsElement);e.classList.remove(yt),e.removeAttribute("aria-current");const i=z.findOne(`[data-bs-slide-to="${t}"]`,this._indicatorsElement);i&&(i.classList.add(yt),i.setAttribute("aria-current","true"))}_updateInterval(){const t=this._activeElement||this._getActive();if(!t)return;const e=Number.parseInt(t.getAttribute("data-bs-interval"),10);this._config.interval=e||this._config.defaultInterval}_slide(t,e=null){if(this._isSliding)return;const i=this._getActive(),n=t===at,s=e||b(this._getItems(),i,n,this._config.wrap);if(s===i)return;const o=this._getItemIndex(s),r=e=>N.trigger(this._element,e,{relatedTarget:s,direction:this._orderToDirection(t),from:this._getItemIndex(i),to:o});if(r(dt).defaultPrevented)return;if(!i||!s)return;const a=Boolean(this._interval);this.pause(),this._isSliding=!0,this._setActiveIndicatorElement(o),this._activeElement=s;const l=n?"carousel-item-start":"carousel-item-end",c=n?"carousel-item-next":"carousel-item-prev";s.classList.add(c),d(s),i.classList.add(l),s.classList.add(l),this._queueCallback((()=>{s.classList.remove(l,c),s.classList.add(yt),i.classList.remove(yt,c,l),this._isSliding=!1,r(ut)}),i,this._isAnimated()),a&&this.cycle()}_isAnimated(){return this._element.classList.contains("slide")}_getActive(){return z.findOne(Et,this._element)}_getItems(){return z.find(At,this._element)}_clearInterval(){this._interval&&(clearInterval(this._interval),this._interval=null)}_directionToOrder(t){return p()?t===ct?lt:at:t===ct?at:lt}_orderToDirection(t){return p()?t===lt?ct:ht:t===lt?ht:ct}static jQueryInterface(t){return this.each((function(){const e=xt.getOrCreateInstance(this,t);if("number"!=typeof t){if("string"==typeof t){if(void 0===e[t]||t.startsWith("_")||"constructor"===t)throw new TypeError(`No method named "${t}"`);e[t]()}}else e.to(t)}))}}N.on(document,bt,"[data-bs-slide], [data-bs-slide-to]",(function(t){const e=z.getElementFromSelector(this);if(!e||!e.classList.contains(vt))return;t.preventDefault();const i=xt.getOrCreateInstance(e),n=this.getAttribute("data-bs-slide-to");return n?(i.to(n),void i._maybeEnableCycle()):"next"===F.getDataAttribute(this,"slide")?(i.next(),void i._maybeEnableCycle()):(i.prev(),void i._maybeEnableCycle())})),N.on(window,_t,(()=>{const t=z.find('[data-bs-ride="carousel"]');for(const e of t)xt.getOrCreateInstance(e)})),m(xt);const kt=".bs.collapse",Lt=`show${kt}`,St=`shown${kt}`,Dt=`hide${kt}`,$t=`hidden${kt}`,It=`click${kt}.data-api`,Nt="show",Pt="collapse",jt="collapsing",Mt=`:scope .${Pt} .${Pt}`,Ft='[data-bs-toggle="collapse"]',Ht={parent:null,toggle:!0},Wt={parent:"(null|element)",toggle:"boolean"};class Bt extends W{constructor(t,e){super(t,e),this._isTransitioning=!1,this._triggerArray=[];const i=z.find(Ft);for(const t of i){const e=z.getSelectorFromElement(t),i=z.find(e).filter((t=>t===this._element));null!==e&&i.length&&this._triggerArray.push(t)}this._initializeChildren(),this._config.parent||this._addAriaAndCollapsedClass(this._triggerArray,this._isShown()),this._config.toggle&&this.toggle()}static get Default(){return Ht}static get DefaultType(){return Wt}static get NAME(){return"collapse"}toggle(){this._isShown()?this.hide():this.show()}show(){if(this._isTransitioning||this._isShown())return;let t=[];if(this._config.parent&&(t=this._getFirstLevelChildren(".collapse.show, .collapse.collapsing").filter((t=>t!==this._element)).map((t=>Bt.getOrCreateInstance(t,{toggle:!1})))),t.length&&t[0]._isTransitioning)return;if(N.trigger(this._element,Lt).defaultPrevented)return;for(const e of t)e.hide();const e=this._getDimension();this._element.classList.remove(Pt),this._element.classList.add(jt),this._element.style[e]=0,this._addAriaAndCollapsedClass(this._triggerArray,!0),this._isTransitioning=!0;const i=`scroll${e[0].toUpperCase()+e.slice(1)}`;this._queueCallback((()=>{this._isTransitioning=!1,this._element.classList.remove(jt),this._element.classList.add(Pt,Nt),this._element.style[e]="",N.trigger(this._element,St)}),this._element,!0),this._element.style[e]=`${this._element[i]}px`}hide(){if(this._isTransitioning||!this._isShown())return;if(N.trigger(this._element,Dt).defaultPrevented)return;const t=this._getDimension();this._element.style[t]=`${this._element.getBoundingClientRect()[t]}px`,d(this._element),this._element.classList.add(jt),this._element.classList.remove(Pt,Nt);for(const t of this._triggerArray){const e=z.getElementFromSelector(t);e&&!this._isShown(e)&&this._addAriaAndCollapsedClass([t],!1)}this._isTransitioning=!0,this._element.style[t]="",this._queueCallback((()=>{this._isTransitioning=!1,this._element.classList.remove(jt),this._element.classList.add(Pt),N.trigger(this._element,$t)}),this._element,!0)}_isShown(t=this._element){return t.classList.contains(Nt)}_configAfterMerge(t){return t.toggle=Boolean(t.toggle),t.parent=r(t.parent),t}_getDimension(){return this._element.classList.contains("collapse-horizontal")?"width":"height"}_initializeChildren(){if(!this._config.parent)return;const t=this._getFirstLevelChildren(Ft);for(const e of t){const t=z.getElementFromSelector(e);t&&this._addAriaAndCollapsedClass([e],this._isShown(t))}}_getFirstLevelChildren(t){const e=z.find(Mt,this._config.parent);return z.find(t,this._config.parent).filter((t=>!e.includes(t)))}_addAriaAndCollapsedClass(t,e){if(t.length)for(const i of t)i.classList.toggle("collapsed",!e),i.setAttribute("aria-expanded",e)}static jQueryInterface(t){const e={};return"string"==typeof t&&/show|hide/.test(t)&&(e.toggle=!1),this.each((function(){const i=Bt.getOrCreateInstance(this,e);if("string"==typeof t){if(void 0===i[t])throw new TypeError(`No method named "${t}"`);i[t]()}}))}}N.on(document,It,Ft,(function(t){("A"===t.target.tagName||t.delegateTarget&&"A"===t.delegateTarget.tagName)&&t.preventDefault();for(const t of z.getMultipleElementsFromSelector(this))Bt.getOrCreateInstance(t,{toggle:!1}).toggle()})),m(Bt);var zt="top",Rt="bottom",qt="right",Vt="left",Kt="auto",Qt=[zt,Rt,qt,Vt],Xt="start",Yt="end",Ut="clippingParents",Gt="viewport",Jt="popper",Zt="reference",te=Qt.reduce((function(t,e){return t.concat([e+"-"+Xt,e+"-"+Yt])}),[]),ee=[].concat(Qt,[Kt]).reduce((function(t,e){return t.concat([e,e+"-"+Xt,e+"-"+Yt])}),[]),ie="beforeRead",ne="read",se="afterRead",oe="beforeMain",re="main",ae="afterMain",le="beforeWrite",ce="write",he="afterWrite",de=[ie,ne,se,oe,re,ae,le,ce,he];function ue(t){return t?(t.nodeName||"").toLowerCase():null}function fe(t){if(null==t)return window;if("[object Window]"!==t.toString()){var e=t.ownerDocument;return e&&e.defaultView||window}return t}function pe(t){return t instanceof fe(t).Element||t instanceof Element}function me(t){return t instanceof fe(t).HTMLElement||t instanceof HTMLElement}function ge(t){return"undefined"!=typeof ShadowRoot&&(t instanceof fe(t).ShadowRoot||t instanceof ShadowRoot)}const _e={name:"applyStyles",enabled:!0,phase:"write",fn:function(t){var e=t.state;Object.keys(e.elements).forEach((function(t){var i=e.styles[t]||{},n=e.attributes[t]||{},s=e.elements[t];me(s)&&ue(s)&&(Object.assign(s.style,i),Object.keys(n).forEach((function(t){var e=n[t];!1===e?s.removeAttribute(t):s.setAttribute(t,!0===e?"":e)})))}))},effect:function(t){var e=t.state,i={popper:{position:e.options.strategy,left:"0",top:"0",margin:"0"},arrow:{position:"absolute"},reference:{}};return Object.assign(e.elements.popper.style,i.popper),e.styles=i,e.elements.arrow&&Object.assign(e.elements.arrow.style,i.arrow),function(){Object.keys(e.elements).forEach((function(t){var n=e.elements[t],s=e.attributes[t]||{},o=Object.keys(e.styles.hasOwnProperty(t)?e.styles[t]:i[t]).reduce((function(t,e){return t[e]="",t}),{});me(n)&&ue(n)&&(Object.assign(n.style,o),Object.keys(s).forEach((function(t){n.removeAttribute(t)})))}))}},requires:["computeStyles"]};function be(t){return t.split("-")[0]}var ve=Math.max,ye=Math.min,we=Math.round;function Ae(){var t=navigator.userAgentData;return null!=t&&t.brands&&Array.isArray(t.brands)?t.brands.map((function(t){return t.brand+"/"+t.version})).join(" "):navigator.userAgent}function Ee(){return!/^((?!chrome|android).)*safari/i.test(Ae())}function Te(t,e,i){void 0===e&&(e=!1),void 0===i&&(i=!1);var n=t.getBoundingClientRect(),s=1,o=1;e&&me(t)&&(s=t.offsetWidth>0&&we(n.width)/t.offsetWidth||1,o=t.offsetHeight>0&&we(n.height)/t.offsetHeight||1);var r=(pe(t)?fe(t):window).visualViewport,a=!Ee()&&i,l=(n.left+(a&&r?r.offsetLeft:0))/s,c=(n.top+(a&&r?r.offsetTop:0))/o,h=n.width/s,d=n.height/o;return{width:h,height:d,top:c,right:l+h,bottom:c+d,left:l,x:l,y:c}}function Ce(t){var e=Te(t),i=t.offsetWidth,n=t.offsetHeight;return Math.abs(e.width-i)<=1&&(i=e.width),Math.abs(e.height-n)<=1&&(n=e.height),{x:t.offsetLeft,y:t.offsetTop,width:i,height:n}}function Oe(t,e){var i=e.getRootNode&&e.getRootNode();if(t.contains(e))return!0;if(i&&ge(i)){var n=e;do{if(n&&t.isSameNode(n))return!0;n=n.parentNode||n.host}while(n)}return!1}function xe(t){return fe(t).getComputedStyle(t)}function ke(t){return["table","td","th"].indexOf(ue(t))>=0}function Le(t){return((pe(t)?t.ownerDocument:t.document)||window.document).documentElement}function Se(t){return"html"===ue(t)?t:t.assignedSlot||t.parentNode||(ge(t)?t.host:null)||Le(t)}function De(t){return me(t)&&"fixed"!==xe(t).position?t.offsetParent:null}function $e(t){for(var e=fe(t),i=De(t);i&&ke(i)&&"static"===xe(i).position;)i=De(i);return i&&("html"===ue(i)||"body"===ue(i)&&"static"===xe(i).position)?e:i||function(t){var e=/firefox/i.test(Ae());if(/Trident/i.test(Ae())&&me(t)&&"fixed"===xe(t).position)return null;var i=Se(t);for(ge(i)&&(i=i.host);me(i)&&["html","body"].indexOf(ue(i))<0;){var n=xe(i);if("none"!==n.transform||"none"!==n.perspective||"paint"===n.contain||-1!==["transform","perspective"].indexOf(n.willChange)||e&&"filter"===n.willChange||e&&n.filter&&"none"!==n.filter)return i;i=i.parentNode}return null}(t)||e}function Ie(t){return["top","bottom"].indexOf(t)>=0?"x":"y"}function Ne(t,e,i){return ve(t,ye(e,i))}function Pe(t){return Object.assign({},{top:0,right:0,bottom:0,left:0},t)}function je(t,e){return e.reduce((function(e,i){return e[i]=t,e}),{})}const Me={name:"arrow",enabled:!0,phase:"main",fn:function(t){var e,i=t.state,n=t.name,s=t.options,o=i.elements.arrow,r=i.modifiersData.popperOffsets,a=be(i.placement),l=Ie(a),c=[Vt,qt].indexOf(a)>=0?"height":"width";if(o&&r){var h=function(t,e){return Pe("number"!=typeof(t="function"==typeof t?t(Object.assign({},e.rects,{placement:e.placement})):t)?t:je(t,Qt))}(s.padding,i),d=Ce(o),u="y"===l?zt:Vt,f="y"===l?Rt:qt,p=i.rects.reference[c]+i.rects.reference[l]-r[l]-i.rects.popper[c],m=r[l]-i.rects.reference[l],g=$e(o),_=g?"y"===l?g.clientHeight||0:g.clientWidth||0:0,b=p/2-m/2,v=h[u],y=_-d[c]-h[f],w=_/2-d[c]/2+b,A=Ne(v,w,y),E=l;i.modifiersData[n]=((e={})[E]=A,e.centerOffset=A-w,e)}},effect:function(t){var e=t.state,i=t.options.element,n=void 0===i?"[data-popper-arrow]":i;null!=n&&("string"!=typeof n||(n=e.elements.popper.querySelector(n)))&&Oe(e.elements.popper,n)&&(e.elements.arrow=n)},requires:["popperOffsets"],requiresIfExists:["preventOverflow"]};function Fe(t){return t.split("-")[1]}var He={top:"auto",right:"auto",bottom:"auto",left:"auto"};function We(t){var e,i=t.popper,n=t.popperRect,s=t.placement,o=t.variation,r=t.offsets,a=t.position,l=t.gpuAcceleration,c=t.adaptive,h=t.roundOffsets,d=t.isFixed,u=r.x,f=void 0===u?0:u,p=r.y,m=void 0===p?0:p,g="function"==typeof h?h({x:f,y:m}):{x:f,y:m};f=g.x,m=g.y;var _=r.hasOwnProperty("x"),b=r.hasOwnProperty("y"),v=Vt,y=zt,w=window;if(c){var A=$e(i),E="clientHeight",T="clientWidth";A===fe(i)&&"static"!==xe(A=Le(i)).position&&"absolute"===a&&(E="scrollHeight",T="scrollWidth"),(s===zt||(s===Vt||s===qt)&&o===Yt)&&(y=Rt,m-=(d&&A===w&&w.visualViewport?w.visualViewport.height:A[E])-n.height,m*=l?1:-1),s!==Vt&&(s!==zt&&s!==Rt||o!==Yt)||(v=qt,f-=(d&&A===w&&w.visualViewport?w.visualViewport.width:A[T])-n.width,f*=l?1:-1)}var C,O=Object.assign({position:a},c&&He),x=!0===h?function(t,e){var i=t.x,n=t.y,s=e.devicePixelRatio||1;return{x:we(i*s)/s||0,y:we(n*s)/s||0}}({x:f,y:m},fe(i)):{x:f,y:m};return f=x.x,m=x.y,l?Object.assign({},O,((C={})[y]=b?"0":"",C[v]=_?"0":"",C.transform=(w.devicePixelRatio||1)<=1?"translate("+f+"px, "+m+"px)":"translate3d("+f+"px, "+m+"px, 0)",C)):Object.assign({},O,((e={})[y]=b?m+"px":"",e[v]=_?f+"px":"",e.transform="",e))}const Be={name:"computeStyles",enabled:!0,phase:"beforeWrite",fn:function(t){var e=t.state,i=t.options,n=i.gpuAcceleration,s=void 0===n||n,o=i.adaptive,r=void 0===o||o,a=i.roundOffsets,l=void 0===a||a,c={placement:be(e.placement),variation:Fe(e.placement),popper:e.elements.popper,popperRect:e.rects.popper,gpuAcceleration:s,isFixed:"fixed"===e.options.strategy};null!=e.modifiersData.popperOffsets&&(e.styles.popper=Object.assign({},e.styles.popper,We(Object.assign({},c,{offsets:e.modifiersData.popperOffsets,position:e.options.strategy,adaptive:r,roundOffsets:l})))),null!=e.modifiersData.arrow&&(e.styles.arrow=Object.assign({},e.styles.arrow,We(Object.assign({},c,{offsets:e.modifiersData.arrow,position:"absolute",adaptive:!1,roundOffsets:l})))),e.attributes.popper=Object.assign({},e.attributes.popper,{"data-popper-placement":e.placement})},data:{}};var ze={passive:!0};const Re={name:"eventListeners",enabled:!0,phase:"write",fn:function(){},effect:function(t){var e=t.state,i=t.instance,n=t.options,s=n.scroll,o=void 0===s||s,r=n.resize,a=void 0===r||r,l=fe(e.elements.popper),c=[].concat(e.scrollParents.reference,e.scrollParents.popper);return o&&c.forEach((function(t){t.addEventListener("scroll",i.update,ze)})),a&&l.addEventListener("resize",i.update,ze),function(){o&&c.forEach((function(t){t.removeEventListener("scroll",i.update,ze)})),a&&l.removeEventListener("resize",i.update,ze)}},data:{}};var qe={left:"right",right:"left",bottom:"top",top:"bottom"};function Ve(t){return t.replace(/left|right|bottom|top/g,(function(t){return qe[t]}))}var Ke={start:"end",end:"start"};function Qe(t){return t.replace(/start|end/g,(function(t){return Ke[t]}))}function Xe(t){var e=fe(t);return{scrollLeft:e.pageXOffset,scrollTop:e.pageYOffset}}function Ye(t){return Te(Le(t)).left+Xe(t).scrollLeft}function Ue(t){var e=xe(t),i=e.overflow,n=e.overflowX,s=e.overflowY;return/auto|scroll|overlay|hidden/.test(i+s+n)}function Ge(t){return["html","body","#document"].indexOf(ue(t))>=0?t.ownerDocument.body:me(t)&&Ue(t)?t:Ge(Se(t))}function Je(t,e){var i;void 0===e&&(e=[]);var n=Ge(t),s=n===(null==(i=t.ownerDocument)?void 0:i.body),o=fe(n),r=s?[o].concat(o.visualViewport||[],Ue(n)?n:[]):n,a=e.concat(r);return s?a:a.concat(Je(Se(r)))}function Ze(t){return Object.assign({},t,{left:t.x,top:t.y,right:t.x+t.width,bottom:t.y+t.height})}function ti(t,e,i){return e===Gt?Ze(function(t,e){var i=fe(t),n=Le(t),s=i.visualViewport,o=n.clientWidth,r=n.clientHeight,a=0,l=0;if(s){o=s.width,r=s.height;var c=Ee();(c||!c&&"fixed"===e)&&(a=s.offsetLeft,l=s.offsetTop)}return{width:o,height:r,x:a+Ye(t),y:l}}(t,i)):pe(e)?function(t,e){var i=Te(t,!1,"fixed"===e);return i.top=i.top+t.clientTop,i.left=i.left+t.clientLeft,i.bottom=i.top+t.clientHeight,i.right=i.left+t.clientWidth,i.width=t.clientWidth,i.height=t.clientHeight,i.x=i.left,i.y=i.top,i}(e,i):Ze(function(t){var e,i=Le(t),n=Xe(t),s=null==(e=t.ownerDocument)?void 0:e.body,o=ve(i.scrollWidth,i.clientWidth,s?s.scrollWidth:0,s?s.clientWidth:0),r=ve(i.scrollHeight,i.clientHeight,s?s.scrollHeight:0,s?s.clientHeight:0),a=-n.scrollLeft+Ye(t),l=-n.scrollTop;return"rtl"===xe(s||i).direction&&(a+=ve(i.clientWidth,s?s.clientWidth:0)-o),{width:o,height:r,x:a,y:l}}(Le(t)))}function ei(t){var e,i=t.reference,n=t.element,s=t.placement,o=s?be(s):null,r=s?Fe(s):null,a=i.x+i.width/2-n.width/2,l=i.y+i.height/2-n.height/2;switch(o){case zt:e={x:a,y:i.y-n.height};break;case Rt:e={x:a,y:i.y+i.height};break;case qt:e={x:i.x+i.width,y:l};break;case Vt:e={x:i.x-n.width,y:l};break;default:e={x:i.x,y:i.y}}var c=o?Ie(o):null;if(null!=c){var h="y"===c?"height":"width";switch(r){case Xt:e[c]=e[c]-(i[h]/2-n[h]/2);break;case Yt:e[c]=e[c]+(i[h]/2-n[h]/2)}}return e}function ii(t,e){void 0===e&&(e={});var i=e,n=i.placement,s=void 0===n?t.placement:n,o=i.strategy,r=void 0===o?t.strategy:o,a=i.boundary,l=void 0===a?Ut:a,c=i.rootBoundary,h=void 0===c?Gt:c,d=i.elementContext,u=void 0===d?Jt:d,f=i.altBoundary,p=void 0!==f&&f,m=i.padding,g=void 0===m?0:m,_=Pe("number"!=typeof g?g:je(g,Qt)),b=u===Jt?Zt:Jt,v=t.rects.popper,y=t.elements[p?b:u],w=function(t,e,i,n){var s="clippingParents"===e?function(t){var e=Je(Se(t)),i=["absolute","fixed"].indexOf(xe(t).position)>=0&&me(t)?$e(t):t;return pe(i)?e.filter((function(t){return pe(t)&&Oe(t,i)&&"body"!==ue(t)})):[]}(t):[].concat(e),o=[].concat(s,[i]),r=o[0],a=o.reduce((function(e,i){var s=ti(t,i,n);return e.top=ve(s.top,e.top),e.right=ye(s.right,e.right),e.bottom=ye(s.bottom,e.bottom),e.left=ve(s.left,e.left),e}),ti(t,r,n));return a.width=a.right-a.left,a.height=a.bottom-a.top,a.x=a.left,a.y=a.top,a}(pe(y)?y:y.contextElement||Le(t.elements.popper),l,h,r),A=Te(t.elements.reference),E=ei({reference:A,element:v,strategy:"absolute",placement:s}),T=Ze(Object.assign({},v,E)),C=u===Jt?T:A,O={top:w.top-C.top+_.top,bottom:C.bottom-w.bottom+_.bottom,left:w.left-C.left+_.left,right:C.right-w.right+_.right},x=t.modifiersData.offset;if(u===Jt&&x){var k=x[s];Object.keys(O).forEach((function(t){var e=[qt,Rt].indexOf(t)>=0?1:-1,i=[zt,Rt].indexOf(t)>=0?"y":"x";O[t]+=k[i]*e}))}return O}function ni(t,e){void 0===e&&(e={});var i=e,n=i.placement,s=i.boundary,o=i.rootBoundary,r=i.padding,a=i.flipVariations,l=i.allowedAutoPlacements,c=void 0===l?ee:l,h=Fe(n),d=h?a?te:te.filter((function(t){return Fe(t)===h})):Qt,u=d.filter((function(t){return c.indexOf(t)>=0}));0===u.length&&(u=d);var f=u.reduce((function(e,i){return e[i]=ii(t,{placement:i,boundary:s,rootBoundary:o,padding:r})[be(i)],e}),{});return Object.keys(f).sort((function(t,e){return f[t]-f[e]}))}const si={name:"flip",enabled:!0,phase:"main",fn:function(t){var e=t.state,i=t.options,n=t.name;if(!e.modifiersData[n]._skip){for(var s=i.mainAxis,o=void 0===s||s,r=i.altAxis,a=void 0===r||r,l=i.fallbackPlacements,c=i.padding,h=i.boundary,d=i.rootBoundary,u=i.altBoundary,f=i.flipVariations,p=void 0===f||f,m=i.allowedAutoPlacements,g=e.options.placement,_=be(g),b=l||(_!==g&&p?function(t){if(be(t)===Kt)return[];var e=Ve(t);return[Qe(t),e,Qe(e)]}(g):[Ve(g)]),v=[g].concat(b).reduce((function(t,i){return t.concat(be(i)===Kt?ni(e,{placement:i,boundary:h,rootBoundary:d,padding:c,flipVariations:p,allowedAutoPlacements:m}):i)}),[]),y=e.rects.reference,w=e.rects.popper,A=new Map,E=!0,T=v[0],C=0;C=0,S=L?"width":"height",D=ii(e,{placement:O,boundary:h,rootBoundary:d,altBoundary:u,padding:c}),$=L?k?qt:Vt:k?Rt:zt;y[S]>w[S]&&($=Ve($));var I=Ve($),N=[];if(o&&N.push(D[x]<=0),a&&N.push(D[$]<=0,D[I]<=0),N.every((function(t){return t}))){T=O,E=!1;break}A.set(O,N)}if(E)for(var P=function(t){var e=v.find((function(e){var i=A.get(e);if(i)return i.slice(0,t).every((function(t){return t}))}));if(e)return T=e,"break"},j=p?3:1;j>0&&"break"!==P(j);j--);e.placement!==T&&(e.modifiersData[n]._skip=!0,e.placement=T,e.reset=!0)}},requiresIfExists:["offset"],data:{_skip:!1}};function oi(t,e,i){return void 0===i&&(i={x:0,y:0}),{top:t.top-e.height-i.y,right:t.right-e.width+i.x,bottom:t.bottom-e.height+i.y,left:t.left-e.width-i.x}}function ri(t){return[zt,qt,Rt,Vt].some((function(e){return t[e]>=0}))}const ai={name:"hide",enabled:!0,phase:"main",requiresIfExists:["preventOverflow"],fn:function(t){var e=t.state,i=t.name,n=e.rects.reference,s=e.rects.popper,o=e.modifiersData.preventOverflow,r=ii(e,{elementContext:"reference"}),a=ii(e,{altBoundary:!0}),l=oi(r,n),c=oi(a,s,o),h=ri(l),d=ri(c);e.modifiersData[i]={referenceClippingOffsets:l,popperEscapeOffsets:c,isReferenceHidden:h,hasPopperEscaped:d},e.attributes.popper=Object.assign({},e.attributes.popper,{"data-popper-reference-hidden":h,"data-popper-escaped":d})}},li={name:"offset",enabled:!0,phase:"main",requires:["popperOffsets"],fn:function(t){var e=t.state,i=t.options,n=t.name,s=i.offset,o=void 0===s?[0,0]:s,r=ee.reduce((function(t,i){return t[i]=function(t,e,i){var n=be(t),s=[Vt,zt].indexOf(n)>=0?-1:1,o="function"==typeof i?i(Object.assign({},e,{placement:t})):i,r=o[0],a=o[1];return r=r||0,a=(a||0)*s,[Vt,qt].indexOf(n)>=0?{x:a,y:r}:{x:r,y:a}}(i,e.rects,o),t}),{}),a=r[e.placement],l=a.x,c=a.y;null!=e.modifiersData.popperOffsets&&(e.modifiersData.popperOffsets.x+=l,e.modifiersData.popperOffsets.y+=c),e.modifiersData[n]=r}},ci={name:"popperOffsets",enabled:!0,phase:"read",fn:function(t){var e=t.state,i=t.name;e.modifiersData[i]=ei({reference:e.rects.reference,element:e.rects.popper,strategy:"absolute",placement:e.placement})},data:{}},hi={name:"preventOverflow",enabled:!0,phase:"main",fn:function(t){var e=t.state,i=t.options,n=t.name,s=i.mainAxis,o=void 0===s||s,r=i.altAxis,a=void 0!==r&&r,l=i.boundary,c=i.rootBoundary,h=i.altBoundary,d=i.padding,u=i.tether,f=void 0===u||u,p=i.tetherOffset,m=void 0===p?0:p,g=ii(e,{boundary:l,rootBoundary:c,padding:d,altBoundary:h}),_=be(e.placement),b=Fe(e.placement),v=!b,y=Ie(_),w="x"===y?"y":"x",A=e.modifiersData.popperOffsets,E=e.rects.reference,T=e.rects.popper,C="function"==typeof m?m(Object.assign({},e.rects,{placement:e.placement})):m,O="number"==typeof C?{mainAxis:C,altAxis:C}:Object.assign({mainAxis:0,altAxis:0},C),x=e.modifiersData.offset?e.modifiersData.offset[e.placement]:null,k={x:0,y:0};if(A){if(o){var L,S="y"===y?zt:Vt,D="y"===y?Rt:qt,$="y"===y?"height":"width",I=A[y],N=I+g[S],P=I-g[D],j=f?-T[$]/2:0,M=b===Xt?E[$]:T[$],F=b===Xt?-T[$]:-E[$],H=e.elements.arrow,W=f&&H?Ce(H):{width:0,height:0},B=e.modifiersData["arrow#persistent"]?e.modifiersData["arrow#persistent"].padding:{top:0,right:0,bottom:0,left:0},z=B[S],R=B[D],q=Ne(0,E[$],W[$]),V=v?E[$]/2-j-q-z-O.mainAxis:M-q-z-O.mainAxis,K=v?-E[$]/2+j+q+R+O.mainAxis:F+q+R+O.mainAxis,Q=e.elements.arrow&&$e(e.elements.arrow),X=Q?"y"===y?Q.clientTop||0:Q.clientLeft||0:0,Y=null!=(L=null==x?void 0:x[y])?L:0,U=I+K-Y,G=Ne(f?ye(N,I+V-Y-X):N,I,f?ve(P,U):P);A[y]=G,k[y]=G-I}if(a){var J,Z="x"===y?zt:Vt,tt="x"===y?Rt:qt,et=A[w],it="y"===w?"height":"width",nt=et+g[Z],st=et-g[tt],ot=-1!==[zt,Vt].indexOf(_),rt=null!=(J=null==x?void 0:x[w])?J:0,at=ot?nt:et-E[it]-T[it]-rt+O.altAxis,lt=ot?et+E[it]+T[it]-rt-O.altAxis:st,ct=f&&ot?function(t,e,i){var n=Ne(t,e,i);return n>i?i:n}(at,et,lt):Ne(f?at:nt,et,f?lt:st);A[w]=ct,k[w]=ct-et}e.modifiersData[n]=k}},requiresIfExists:["offset"]};function di(t,e,i){void 0===i&&(i=!1);var n,s,o=me(e),r=me(e)&&function(t){var e=t.getBoundingClientRect(),i=we(e.width)/t.offsetWidth||1,n=we(e.height)/t.offsetHeight||1;return 1!==i||1!==n}(e),a=Le(e),l=Te(t,r,i),c={scrollLeft:0,scrollTop:0},h={x:0,y:0};return(o||!o&&!i)&&(("body"!==ue(e)||Ue(a))&&(c=(n=e)!==fe(n)&&me(n)?{scrollLeft:(s=n).scrollLeft,scrollTop:s.scrollTop}:Xe(n)),me(e)?((h=Te(e,!0)).x+=e.clientLeft,h.y+=e.clientTop):a&&(h.x=Ye(a))),{x:l.left+c.scrollLeft-h.x,y:l.top+c.scrollTop-h.y,width:l.width,height:l.height}}function ui(t){var e=new Map,i=new Set,n=[];function s(t){i.add(t.name),[].concat(t.requires||[],t.requiresIfExists||[]).forEach((function(t){if(!i.has(t)){var n=e.get(t);n&&s(n)}})),n.push(t)}return t.forEach((function(t){e.set(t.name,t)})),t.forEach((function(t){i.has(t.name)||s(t)})),n}var fi={placement:"bottom",modifiers:[],strategy:"absolute"};function pi(){for(var t=arguments.length,e=new Array(t),i=0;iNumber.parseInt(t,10))):"function"==typeof t?e=>t(e,this._element):t}_getPopperConfig(){const t={placement:this._getPlacement(),modifiers:[{name:"preventOverflow",options:{boundary:this._config.boundary}},{name:"offset",options:{offset:this._getOffset()}}]};return(this._inNavbar||"static"===this._config.display)&&(F.setDataAttribute(this._menu,"popper","static"),t.modifiers=[{name:"applyStyles",enabled:!1}]),{...t,...g(this._config.popperConfig,[t])}}_selectMenuItem({key:t,target:e}){const i=z.find(".dropdown-menu .dropdown-item:not(.disabled):not(:disabled)",this._menu).filter((t=>a(t)));i.length&&b(i,e,t===Ti,!i.includes(e)).focus()}static jQueryInterface(t){return this.each((function(){const e=qi.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t])throw new TypeError(`No method named "${t}"`);e[t]()}}))}static clearMenus(t){if(2===t.button||"keyup"===t.type&&"Tab"!==t.key)return;const e=z.find(Ni);for(const i of e){const e=qi.getInstance(i);if(!e||!1===e._config.autoClose)continue;const n=t.composedPath(),s=n.includes(e._menu);if(n.includes(e._element)||"inside"===e._config.autoClose&&!s||"outside"===e._config.autoClose&&s)continue;if(e._menu.contains(t.target)&&("keyup"===t.type&&"Tab"===t.key||/input|select|option|textarea|form/i.test(t.target.tagName)))continue;const o={relatedTarget:e._element};"click"===t.type&&(o.clickEvent=t),e._completeHide(o)}}static dataApiKeydownHandler(t){const e=/input|textarea/i.test(t.target.tagName),i="Escape"===t.key,n=[Ei,Ti].includes(t.key);if(!n&&!i)return;if(e&&!i)return;t.preventDefault();const s=this.matches(Ii)?this:z.prev(this,Ii)[0]||z.next(this,Ii)[0]||z.findOne(Ii,t.delegateTarget.parentNode),o=qi.getOrCreateInstance(s);if(n)return t.stopPropagation(),o.show(),void o._selectMenuItem(t);o._isShown()&&(t.stopPropagation(),o.hide(),s.focus())}}N.on(document,Si,Ii,qi.dataApiKeydownHandler),N.on(document,Si,Pi,qi.dataApiKeydownHandler),N.on(document,Li,qi.clearMenus),N.on(document,Di,qi.clearMenus),N.on(document,Li,Ii,(function(t){t.preventDefault(),qi.getOrCreateInstance(this).toggle()})),m(qi);const Vi="backdrop",Ki="show",Qi=`mousedown.bs.${Vi}`,Xi={className:"modal-backdrop",clickCallback:null,isAnimated:!1,isVisible:!0,rootElement:"body"},Yi={className:"string",clickCallback:"(function|null)",isAnimated:"boolean",isVisible:"boolean",rootElement:"(element|string)"};class Ui extends H{constructor(t){super(),this._config=this._getConfig(t),this._isAppended=!1,this._element=null}static get Default(){return Xi}static get DefaultType(){return Yi}static get NAME(){return Vi}show(t){if(!this._config.isVisible)return void g(t);this._append();const e=this._getElement();this._config.isAnimated&&d(e),e.classList.add(Ki),this._emulateAnimation((()=>{g(t)}))}hide(t){this._config.isVisible?(this._getElement().classList.remove(Ki),this._emulateAnimation((()=>{this.dispose(),g(t)}))):g(t)}dispose(){this._isAppended&&(N.off(this._element,Qi),this._element.remove(),this._isAppended=!1)}_getElement(){if(!this._element){const t=document.createElement("div");t.className=this._config.className,this._config.isAnimated&&t.classList.add("fade"),this._element=t}return this._element}_configAfterMerge(t){return t.rootElement=r(t.rootElement),t}_append(){if(this._isAppended)return;const t=this._getElement();this._config.rootElement.append(t),N.on(t,Qi,(()=>{g(this._config.clickCallback)})),this._isAppended=!0}_emulateAnimation(t){_(t,this._getElement(),this._config.isAnimated)}}const Gi=".bs.focustrap",Ji=`focusin${Gi}`,Zi=`keydown.tab${Gi}`,tn="backward",en={autofocus:!0,trapElement:null},nn={autofocus:"boolean",trapElement:"element"};class sn extends H{constructor(t){super(),this._config=this._getConfig(t),this._isActive=!1,this._lastTabNavDirection=null}static get Default(){return en}static get DefaultType(){return nn}static get NAME(){return"focustrap"}activate(){this._isActive||(this._config.autofocus&&this._config.trapElement.focus(),N.off(document,Gi),N.on(document,Ji,(t=>this._handleFocusin(t))),N.on(document,Zi,(t=>this._handleKeydown(t))),this._isActive=!0)}deactivate(){this._isActive&&(this._isActive=!1,N.off(document,Gi))}_handleFocusin(t){const{trapElement:e}=this._config;if(t.target===document||t.target===e||e.contains(t.target))return;const i=z.focusableChildren(e);0===i.length?e.focus():this._lastTabNavDirection===tn?i[i.length-1].focus():i[0].focus()}_handleKeydown(t){"Tab"===t.key&&(this._lastTabNavDirection=t.shiftKey?tn:"forward")}}const on=".fixed-top, .fixed-bottom, .is-fixed, .sticky-top",rn=".sticky-top",an="padding-right",ln="margin-right";class cn{constructor(){this._element=document.body}getWidth(){const t=document.documentElement.clientWidth;return Math.abs(window.innerWidth-t)}hide(){const t=this.getWidth();this._disableOverFlow(),this._setElementAttributes(this._element,an,(e=>e+t)),this._setElementAttributes(on,an,(e=>e+t)),this._setElementAttributes(rn,ln,(e=>e-t))}reset(){this._resetElementAttributes(this._element,"overflow"),this._resetElementAttributes(this._element,an),this._resetElementAttributes(on,an),this._resetElementAttributes(rn,ln)}isOverflowing(){return this.getWidth()>0}_disableOverFlow(){this._saveInitialAttribute(this._element,"overflow"),this._element.style.overflow="hidden"}_setElementAttributes(t,e,i){const n=this.getWidth();this._applyManipulationCallback(t,(t=>{if(t!==this._element&&window.innerWidth>t.clientWidth+n)return;this._saveInitialAttribute(t,e);const s=window.getComputedStyle(t).getPropertyValue(e);t.style.setProperty(e,`${i(Number.parseFloat(s))}px`)}))}_saveInitialAttribute(t,e){const i=t.style.getPropertyValue(e);i&&F.setDataAttribute(t,e,i)}_resetElementAttributes(t,e){this._applyManipulationCallback(t,(t=>{const i=F.getDataAttribute(t,e);null!==i?(F.removeDataAttribute(t,e),t.style.setProperty(e,i)):t.style.removeProperty(e)}))}_applyManipulationCallback(t,e){if(o(t))e(t);else for(const i of z.find(t,this._element))e(i)}}const hn=".bs.modal",dn=`hide${hn}`,un=`hidePrevented${hn}`,fn=`hidden${hn}`,pn=`show${hn}`,mn=`shown${hn}`,gn=`resize${hn}`,_n=`click.dismiss${hn}`,bn=`mousedown.dismiss${hn}`,vn=`keydown.dismiss${hn}`,yn=`click${hn}.data-api`,wn="modal-open",An="show",En="modal-static",Tn={backdrop:!0,focus:!0,keyboard:!0},Cn={backdrop:"(boolean|string)",focus:"boolean",keyboard:"boolean"};class On extends W{constructor(t,e){super(t,e),this._dialog=z.findOne(".modal-dialog",this._element),this._backdrop=this._initializeBackDrop(),this._focustrap=this._initializeFocusTrap(),this._isShown=!1,this._isTransitioning=!1,this._scrollBar=new cn,this._addEventListeners()}static get Default(){return Tn}static get DefaultType(){return Cn}static get NAME(){return"modal"}toggle(t){return this._isShown?this.hide():this.show(t)}show(t){this._isShown||this._isTransitioning||N.trigger(this._element,pn,{relatedTarget:t}).defaultPrevented||(this._isShown=!0,this._isTransitioning=!0,this._scrollBar.hide(),document.body.classList.add(wn),this._adjustDialog(),this._backdrop.show((()=>this._showElement(t))))}hide(){this._isShown&&!this._isTransitioning&&(N.trigger(this._element,dn).defaultPrevented||(this._isShown=!1,this._isTransitioning=!0,this._focustrap.deactivate(),this._element.classList.remove(An),this._queueCallback((()=>this._hideModal()),this._element,this._isAnimated())))}dispose(){N.off(window,hn),N.off(this._dialog,hn),this._backdrop.dispose(),this._focustrap.deactivate(),super.dispose()}handleUpdate(){this._adjustDialog()}_initializeBackDrop(){return new Ui({isVisible:Boolean(this._config.backdrop),isAnimated:this._isAnimated()})}_initializeFocusTrap(){return new sn({trapElement:this._element})}_showElement(t){document.body.contains(this._element)||document.body.append(this._element),this._element.style.display="block",this._element.removeAttribute("aria-hidden"),this._element.setAttribute("aria-modal",!0),this._element.setAttribute("role","dialog"),this._element.scrollTop=0;const e=z.findOne(".modal-body",this._dialog);e&&(e.scrollTop=0),d(this._element),this._element.classList.add(An),this._queueCallback((()=>{this._config.focus&&this._focustrap.activate(),this._isTransitioning=!1,N.trigger(this._element,mn,{relatedTarget:t})}),this._dialog,this._isAnimated())}_addEventListeners(){N.on(this._element,vn,(t=>{"Escape"===t.key&&(this._config.keyboard?this.hide():this._triggerBackdropTransition())})),N.on(window,gn,(()=>{this._isShown&&!this._isTransitioning&&this._adjustDialog()})),N.on(this._element,bn,(t=>{N.one(this._element,_n,(e=>{this._element===t.target&&this._element===e.target&&("static"!==this._config.backdrop?this._config.backdrop&&this.hide():this._triggerBackdropTransition())}))}))}_hideModal(){this._element.style.display="none",this._element.setAttribute("aria-hidden",!0),this._element.removeAttribute("aria-modal"),this._element.removeAttribute("role"),this._isTransitioning=!1,this._backdrop.hide((()=>{document.body.classList.remove(wn),this._resetAdjustments(),this._scrollBar.reset(),N.trigger(this._element,fn)}))}_isAnimated(){return this._element.classList.contains("fade")}_triggerBackdropTransition(){if(N.trigger(this._element,un).defaultPrevented)return;const t=this._element.scrollHeight>document.documentElement.clientHeight,e=this._element.style.overflowY;"hidden"===e||this._element.classList.contains(En)||(t||(this._element.style.overflowY="hidden"),this._element.classList.add(En),this._queueCallback((()=>{this._element.classList.remove(En),this._queueCallback((()=>{this._element.style.overflowY=e}),this._dialog)}),this._dialog),this._element.focus())}_adjustDialog(){const t=this._element.scrollHeight>document.documentElement.clientHeight,e=this._scrollBar.getWidth(),i=e>0;if(i&&!t){const t=p()?"paddingLeft":"paddingRight";this._element.style[t]=`${e}px`}if(!i&&t){const t=p()?"paddingRight":"paddingLeft";this._element.style[t]=`${e}px`}}_resetAdjustments(){this._element.style.paddingLeft="",this._element.style.paddingRight=""}static jQueryInterface(t,e){return this.each((function(){const i=On.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===i[t])throw new TypeError(`No method named "${t}"`);i[t](e)}}))}}N.on(document,yn,'[data-bs-toggle="modal"]',(function(t){const e=z.getElementFromSelector(this);["A","AREA"].includes(this.tagName)&&t.preventDefault(),N.one(e,pn,(t=>{t.defaultPrevented||N.one(e,fn,(()=>{a(this)&&this.focus()}))}));const i=z.findOne(".modal.show");i&&On.getInstance(i).hide(),On.getOrCreateInstance(e).toggle(this)})),R(On),m(On);const xn=".bs.offcanvas",kn=".data-api",Ln=`load${xn}${kn}`,Sn="show",Dn="showing",$n="hiding",In=".offcanvas.show",Nn=`show${xn}`,Pn=`shown${xn}`,jn=`hide${xn}`,Mn=`hidePrevented${xn}`,Fn=`hidden${xn}`,Hn=`resize${xn}`,Wn=`click${xn}${kn}`,Bn=`keydown.dismiss${xn}`,zn={backdrop:!0,keyboard:!0,scroll:!1},Rn={backdrop:"(boolean|string)",keyboard:"boolean",scroll:"boolean"};class qn extends W{constructor(t,e){super(t,e),this._isShown=!1,this._backdrop=this._initializeBackDrop(),this._focustrap=this._initializeFocusTrap(),this._addEventListeners()}static get Default(){return zn}static get DefaultType(){return Rn}static get NAME(){return"offcanvas"}toggle(t){return this._isShown?this.hide():this.show(t)}show(t){this._isShown||N.trigger(this._element,Nn,{relatedTarget:t}).defaultPrevented||(this._isShown=!0,this._backdrop.show(),this._config.scroll||(new cn).hide(),this._element.setAttribute("aria-modal",!0),this._element.setAttribute("role","dialog"),this._element.classList.add(Dn),this._queueCallback((()=>{this._config.scroll&&!this._config.backdrop||this._focustrap.activate(),this._element.classList.add(Sn),this._element.classList.remove(Dn),N.trigger(this._element,Pn,{relatedTarget:t})}),this._element,!0))}hide(){this._isShown&&(N.trigger(this._element,jn).defaultPrevented||(this._focustrap.deactivate(),this._element.blur(),this._isShown=!1,this._element.classList.add($n),this._backdrop.hide(),this._queueCallback((()=>{this._element.classList.remove(Sn,$n),this._element.removeAttribute("aria-modal"),this._element.removeAttribute("role"),this._config.scroll||(new cn).reset(),N.trigger(this._element,Fn)}),this._element,!0)))}dispose(){this._backdrop.dispose(),this._focustrap.deactivate(),super.dispose()}_initializeBackDrop(){const t=Boolean(this._config.backdrop);return new Ui({className:"offcanvas-backdrop",isVisible:t,isAnimated:!0,rootElement:this._element.parentNode,clickCallback:t?()=>{"static"!==this._config.backdrop?this.hide():N.trigger(this._element,Mn)}:null})}_initializeFocusTrap(){return new sn({trapElement:this._element})}_addEventListeners(){N.on(this._element,Bn,(t=>{"Escape"===t.key&&(this._config.keyboard?this.hide():N.trigger(this._element,Mn))}))}static jQueryInterface(t){return this.each((function(){const e=qn.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t]||t.startsWith("_")||"constructor"===t)throw new TypeError(`No method named "${t}"`);e[t](this)}}))}}N.on(document,Wn,'[data-bs-toggle="offcanvas"]',(function(t){const e=z.getElementFromSelector(this);if(["A","AREA"].includes(this.tagName)&&t.preventDefault(),l(this))return;N.one(e,Fn,(()=>{a(this)&&this.focus()}));const i=z.findOne(In);i&&i!==e&&qn.getInstance(i).hide(),qn.getOrCreateInstance(e).toggle(this)})),N.on(window,Ln,(()=>{for(const t of z.find(In))qn.getOrCreateInstance(t).show()})),N.on(window,Hn,(()=>{for(const t of z.find("[aria-modal][class*=show][class*=offcanvas-]"))"fixed"!==getComputedStyle(t).position&&qn.getOrCreateInstance(t).hide()})),R(qn),m(qn);const Vn={"*":["class","dir","id","lang","role",/^aria-[\w-]*$/i],a:["target","href","title","rel"],area:[],b:[],br:[],col:[],code:[],dd:[],div:[],dl:[],dt:[],em:[],hr:[],h1:[],h2:[],h3:[],h4:[],h5:[],h6:[],i:[],img:["src","srcset","alt","title","width","height"],li:[],ol:[],p:[],pre:[],s:[],small:[],span:[],sub:[],sup:[],strong:[],u:[],ul:[]},Kn=new Set(["background","cite","href","itemtype","longdesc","poster","src","xlink:href"]),Qn=/^(?!javascript:)(?:[a-z0-9+.-]+:|[^&:/?#]*(?:[/?#]|$))/i,Xn=(t,e)=>{const i=t.nodeName.toLowerCase();return e.includes(i)?!Kn.has(i)||Boolean(Qn.test(t.nodeValue)):e.filter((t=>t instanceof RegExp)).some((t=>t.test(i)))},Yn={allowList:Vn,content:{},extraClass:"",html:!1,sanitize:!0,sanitizeFn:null,template:"
"},Un={allowList:"object",content:"object",extraClass:"(string|function)",html:"boolean",sanitize:"boolean",sanitizeFn:"(null|function)",template:"string"},Gn={entry:"(string|element|function|null)",selector:"(string|element)"};class Jn extends H{constructor(t){super(),this._config=this._getConfig(t)}static get Default(){return Yn}static get DefaultType(){return Un}static get NAME(){return"TemplateFactory"}getContent(){return Object.values(this._config.content).map((t=>this._resolvePossibleFunction(t))).filter(Boolean)}hasContent(){return this.getContent().length>0}changeContent(t){return this._checkContent(t),this._config.content={...this._config.content,...t},this}toHtml(){const t=document.createElement("div");t.innerHTML=this._maybeSanitize(this._config.template);for(const[e,i]of Object.entries(this._config.content))this._setContent(t,i,e);const e=t.children[0],i=this._resolvePossibleFunction(this._config.extraClass);return i&&e.classList.add(...i.split(" ")),e}_typeCheckConfig(t){super._typeCheckConfig(t),this._checkContent(t.content)}_checkContent(t){for(const[e,i]of Object.entries(t))super._typeCheckConfig({selector:e,entry:i},Gn)}_setContent(t,e,i){const n=z.findOne(i,t);n&&((e=this._resolvePossibleFunction(e))?o(e)?this._putElementInTemplate(r(e),n):this._config.html?n.innerHTML=this._maybeSanitize(e):n.textContent=e:n.remove())}_maybeSanitize(t){return this._config.sanitize?function(t,e,i){if(!t.length)return t;if(i&&"function"==typeof i)return i(t);const n=(new window.DOMParser).parseFromString(t,"text/html"),s=[].concat(...n.body.querySelectorAll("*"));for(const t of s){const i=t.nodeName.toLowerCase();if(!Object.keys(e).includes(i)){t.remove();continue}const n=[].concat(...t.attributes),s=[].concat(e["*"]||[],e[i]||[]);for(const e of n)Xn(e,s)||t.removeAttribute(e.nodeName)}return n.body.innerHTML}(t,this._config.allowList,this._config.sanitizeFn):t}_resolvePossibleFunction(t){return g(t,[this])}_putElementInTemplate(t,e){if(this._config.html)return e.innerHTML="",void e.append(t);e.textContent=t.textContent}}const Zn=new Set(["sanitize","allowList","sanitizeFn"]),ts="fade",es="show",is=".modal",ns="hide.bs.modal",ss="hover",os="focus",rs={AUTO:"auto",TOP:"top",RIGHT:p()?"left":"right",BOTTOM:"bottom",LEFT:p()?"right":"left"},as={allowList:Vn,animation:!0,boundary:"clippingParents",container:!1,customClass:"",delay:0,fallbackPlacements:["top","right","bottom","left"],html:!1,offset:[0,6],placement:"top",popperConfig:null,sanitize:!0,sanitizeFn:null,selector:!1,template:'',title:"",trigger:"hover focus"},ls={allowList:"object",animation:"boolean",boundary:"(string|element)",container:"(string|element|boolean)",customClass:"(string|function)",delay:"(number|object)",fallbackPlacements:"array",html:"boolean",offset:"(array|string|function)",placement:"(string|function)",popperConfig:"(null|object|function)",sanitize:"boolean",sanitizeFn:"(null|function)",selector:"(string|boolean)",template:"string",title:"(string|element|function)",trigger:"string"};class cs extends W{constructor(t,e){if(void 0===vi)throw new TypeError("Bootstrap's tooltips require Popper (https://popper.js.org)");super(t,e),this._isEnabled=!0,this._timeout=0,this._isHovered=null,this._activeTrigger={},this._popper=null,this._templateFactory=null,this._newContent=null,this.tip=null,this._setListeners(),this._config.selector||this._fixTitle()}static get Default(){return as}static get DefaultType(){return ls}static get NAME(){return"tooltip"}enable(){this._isEnabled=!0}disable(){this._isEnabled=!1}toggleEnabled(){this._isEnabled=!this._isEnabled}toggle(){this._isEnabled&&(this._activeTrigger.click=!this._activeTrigger.click,this._isShown()?this._leave():this._enter())}dispose(){clearTimeout(this._timeout),N.off(this._element.closest(is),ns,this._hideModalHandler),this._element.getAttribute("data-bs-original-title")&&this._element.setAttribute("title",this._element.getAttribute("data-bs-original-title")),this._disposePopper(),super.dispose()}show(){if("none"===this._element.style.display)throw new Error("Please use show on visible elements");if(!this._isWithContent()||!this._isEnabled)return;const t=N.trigger(this._element,this.constructor.eventName("show")),e=(c(this._element)||this._element.ownerDocument.documentElement).contains(this._element);if(t.defaultPrevented||!e)return;this._disposePopper();const i=this._getTipElement();this._element.setAttribute("aria-describedby",i.getAttribute("id"));const{container:n}=this._config;if(this._element.ownerDocument.documentElement.contains(this.tip)||(n.append(i),N.trigger(this._element,this.constructor.eventName("inserted"))),this._popper=this._createPopper(i),i.classList.add(es),"ontouchstart"in document.documentElement)for(const t of[].concat(...document.body.children))N.on(t,"mouseover",h);this._queueCallback((()=>{N.trigger(this._element,this.constructor.eventName("shown")),!1===this._isHovered&&this._leave(),this._isHovered=!1}),this.tip,this._isAnimated())}hide(){if(this._isShown()&&!N.trigger(this._element,this.constructor.eventName("hide")).defaultPrevented){if(this._getTipElement().classList.remove(es),"ontouchstart"in document.documentElement)for(const t of[].concat(...document.body.children))N.off(t,"mouseover",h);this._activeTrigger.click=!1,this._activeTrigger[os]=!1,this._activeTrigger[ss]=!1,this._isHovered=null,this._queueCallback((()=>{this._isWithActiveTrigger()||(this._isHovered||this._disposePopper(),this._element.removeAttribute("aria-describedby"),N.trigger(this._element,this.constructor.eventName("hidden")))}),this.tip,this._isAnimated())}}update(){this._popper&&this._popper.update()}_isWithContent(){return Boolean(this._getTitle())}_getTipElement(){return this.tip||(this.tip=this._createTipElement(this._newContent||this._getContentForTemplate())),this.tip}_createTipElement(t){const e=this._getTemplateFactory(t).toHtml();if(!e)return null;e.classList.remove(ts,es),e.classList.add(`bs-${this.constructor.NAME}-auto`);const i=(t=>{do{t+=Math.floor(1e6*Math.random())}while(document.getElementById(t));return t})(this.constructor.NAME).toString();return e.setAttribute("id",i),this._isAnimated()&&e.classList.add(ts),e}setContent(t){this._newContent=t,this._isShown()&&(this._disposePopper(),this.show())}_getTemplateFactory(t){return this._templateFactory?this._templateFactory.changeContent(t):this._templateFactory=new Jn({...this._config,content:t,extraClass:this._resolvePossibleFunction(this._config.customClass)}),this._templateFactory}_getContentForTemplate(){return{".tooltip-inner":this._getTitle()}}_getTitle(){return this._resolvePossibleFunction(this._config.title)||this._element.getAttribute("data-bs-original-title")}_initializeOnDelegatedTarget(t){return this.constructor.getOrCreateInstance(t.delegateTarget,this._getDelegateConfig())}_isAnimated(){return this._config.animation||this.tip&&this.tip.classList.contains(ts)}_isShown(){return this.tip&&this.tip.classList.contains(es)}_createPopper(t){const e=g(this._config.placement,[this,t,this._element]),i=rs[e.toUpperCase()];return bi(this._element,t,this._getPopperConfig(i))}_getOffset(){const{offset:t}=this._config;return"string"==typeof t?t.split(",").map((t=>Number.parseInt(t,10))):"function"==typeof t?e=>t(e,this._element):t}_resolvePossibleFunction(t){return g(t,[this._element])}_getPopperConfig(t){const e={placement:t,modifiers:[{name:"flip",options:{fallbackPlacements:this._config.fallbackPlacements}},{name:"offset",options:{offset:this._getOffset()}},{name:"preventOverflow",options:{boundary:this._config.boundary}},{name:"arrow",options:{element:`.${this.constructor.NAME}-arrow`}},{name:"preSetPlacement",enabled:!0,phase:"beforeMain",fn:t=>{this._getTipElement().setAttribute("data-popper-placement",t.state.placement)}}]};return{...e,...g(this._config.popperConfig,[e])}}_setListeners(){const t=this._config.trigger.split(" ");for(const e of t)if("click"===e)N.on(this._element,this.constructor.eventName("click"),this._config.selector,(t=>{this._initializeOnDelegatedTarget(t).toggle()}));else if("manual"!==e){const t=e===ss?this.constructor.eventName("mouseenter"):this.constructor.eventName("focusin"),i=e===ss?this.constructor.eventName("mouseleave"):this.constructor.eventName("focusout");N.on(this._element,t,this._config.selector,(t=>{const e=this._initializeOnDelegatedTarget(t);e._activeTrigger["focusin"===t.type?os:ss]=!0,e._enter()})),N.on(this._element,i,this._config.selector,(t=>{const e=this._initializeOnDelegatedTarget(t);e._activeTrigger["focusout"===t.type?os:ss]=e._element.contains(t.relatedTarget),e._leave()}))}this._hideModalHandler=()=>{this._element&&this.hide()},N.on(this._element.closest(is),ns,this._hideModalHandler)}_fixTitle(){const t=this._element.getAttribute("title");t&&(this._element.getAttribute("aria-label")||this._element.textContent.trim()||this._element.setAttribute("aria-label",t),this._element.setAttribute("data-bs-original-title",t),this._element.removeAttribute("title"))}_enter(){this._isShown()||this._isHovered?this._isHovered=!0:(this._isHovered=!0,this._setTimeout((()=>{this._isHovered&&this.show()}),this._config.delay.show))}_leave(){this._isWithActiveTrigger()||(this._isHovered=!1,this._setTimeout((()=>{this._isHovered||this.hide()}),this._config.delay.hide))}_setTimeout(t,e){clearTimeout(this._timeout),this._timeout=setTimeout(t,e)}_isWithActiveTrigger(){return Object.values(this._activeTrigger).includes(!0)}_getConfig(t){const e=F.getDataAttributes(this._element);for(const t of Object.keys(e))Zn.has(t)&&delete e[t];return t={...e,..."object"==typeof t&&t?t:{}},t=this._mergeConfigObj(t),t=this._configAfterMerge(t),this._typeCheckConfig(t),t}_configAfterMerge(t){return t.container=!1===t.container?document.body:r(t.container),"number"==typeof t.delay&&(t.delay={show:t.delay,hide:t.delay}),"number"==typeof t.title&&(t.title=t.title.toString()),"number"==typeof t.content&&(t.content=t.content.toString()),t}_getDelegateConfig(){const t={};for(const[e,i]of Object.entries(this._config))this.constructor.Default[e]!==i&&(t[e]=i);return t.selector=!1,t.trigger="manual",t}_disposePopper(){this._popper&&(this._popper.destroy(),this._popper=null),this.tip&&(this.tip.remove(),this.tip=null)}static jQueryInterface(t){return this.each((function(){const e=cs.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t])throw new TypeError(`No method named "${t}"`);e[t]()}}))}}m(cs);const hs={...cs.Default,content:"",offset:[0,8],placement:"right",template:'',trigger:"click"},ds={...cs.DefaultType,content:"(null|string|element|function)"};class us extends cs{static get Default(){return hs}static get DefaultType(){return ds}static get NAME(){return"popover"}_isWithContent(){return this._getTitle()||this._getContent()}_getContentForTemplate(){return{".popover-header":this._getTitle(),".popover-body":this._getContent()}}_getContent(){return this._resolvePossibleFunction(this._config.content)}static jQueryInterface(t){return this.each((function(){const e=us.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t])throw new TypeError(`No method named "${t}"`);e[t]()}}))}}m(us);const fs=".bs.scrollspy",ps=`activate${fs}`,ms=`click${fs}`,gs=`load${fs}.data-api`,_s="active",bs="[href]",vs=".nav-link",ys=`${vs}, .nav-item > ${vs}, .list-group-item`,ws={offset:null,rootMargin:"0px 0px -25%",smoothScroll:!1,target:null,threshold:[.1,.5,1]},As={offset:"(number|null)",rootMargin:"string",smoothScroll:"boolean",target:"element",threshold:"array"};class Es extends W{constructor(t,e){super(t,e),this._targetLinks=new Map,this._observableSections=new Map,this._rootElement="visible"===getComputedStyle(this._element).overflowY?null:this._element,this._activeTarget=null,this._observer=null,this._previousScrollData={visibleEntryTop:0,parentScrollTop:0},this.refresh()}static get Default(){return ws}static get DefaultType(){return As}static get NAME(){return"scrollspy"}refresh(){this._initializeTargetsAndObservables(),this._maybeEnableSmoothScroll(),this._observer?this._observer.disconnect():this._observer=this._getNewObserver();for(const t of this._observableSections.values())this._observer.observe(t)}dispose(){this._observer.disconnect(),super.dispose()}_configAfterMerge(t){return t.target=r(t.target)||document.body,t.rootMargin=t.offset?`${t.offset}px 0px -30%`:t.rootMargin,"string"==typeof t.threshold&&(t.threshold=t.threshold.split(",").map((t=>Number.parseFloat(t)))),t}_maybeEnableSmoothScroll(){this._config.smoothScroll&&(N.off(this._config.target,ms),N.on(this._config.target,ms,bs,(t=>{const e=this._observableSections.get(t.target.hash);if(e){t.preventDefault();const i=this._rootElement||window,n=e.offsetTop-this._element.offsetTop;if(i.scrollTo)return void i.scrollTo({top:n,behavior:"smooth"});i.scrollTop=n}})))}_getNewObserver(){const t={root:this._rootElement,threshold:this._config.threshold,rootMargin:this._config.rootMargin};return new IntersectionObserver((t=>this._observerCallback(t)),t)}_observerCallback(t){const e=t=>this._targetLinks.get(`#${t.target.id}`),i=t=>{this._previousScrollData.visibleEntryTop=t.target.offsetTop,this._process(e(t))},n=(this._rootElement||document.documentElement).scrollTop,s=n>=this._previousScrollData.parentScrollTop;this._previousScrollData.parentScrollTop=n;for(const o of t){if(!o.isIntersecting){this._activeTarget=null,this._clearActiveClass(e(o));continue}const t=o.target.offsetTop>=this._previousScrollData.visibleEntryTop;if(s&&t){if(i(o),!n)return}else s||t||i(o)}}_initializeTargetsAndObservables(){this._targetLinks=new Map,this._observableSections=new Map;const t=z.find(bs,this._config.target);for(const e of t){if(!e.hash||l(e))continue;const t=z.findOne(decodeURI(e.hash),this._element);a(t)&&(this._targetLinks.set(decodeURI(e.hash),e),this._observableSections.set(e.hash,t))}}_process(t){this._activeTarget!==t&&(this._clearActiveClass(this._config.target),this._activeTarget=t,t.classList.add(_s),this._activateParents(t),N.trigger(this._element,ps,{relatedTarget:t}))}_activateParents(t){if(t.classList.contains("dropdown-item"))z.findOne(".dropdown-toggle",t.closest(".dropdown")).classList.add(_s);else for(const e of z.parents(t,".nav, .list-group"))for(const t of z.prev(e,ys))t.classList.add(_s)}_clearActiveClass(t){t.classList.remove(_s);const e=z.find(`${bs}.${_s}`,t);for(const t of e)t.classList.remove(_s)}static jQueryInterface(t){return this.each((function(){const e=Es.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t]||t.startsWith("_")||"constructor"===t)throw new TypeError(`No method named "${t}"`);e[t]()}}))}}N.on(window,gs,(()=>{for(const t of z.find('[data-bs-spy="scroll"]'))Es.getOrCreateInstance(t)})),m(Es);const Ts=".bs.tab",Cs=`hide${Ts}`,Os=`hidden${Ts}`,xs=`show${Ts}`,ks=`shown${Ts}`,Ls=`click${Ts}`,Ss=`keydown${Ts}`,Ds=`load${Ts}`,$s="ArrowLeft",Is="ArrowRight",Ns="ArrowUp",Ps="ArrowDown",js="Home",Ms="End",Fs="active",Hs="fade",Ws="show",Bs=".dropdown-toggle",zs=`:not(${Bs})`,Rs='[data-bs-toggle="tab"], [data-bs-toggle="pill"], [data-bs-toggle="list"]',qs=`.nav-link${zs}, .list-group-item${zs}, [role="tab"]${zs}, ${Rs}`,Vs=`.${Fs}[data-bs-toggle="tab"], .${Fs}[data-bs-toggle="pill"], .${Fs}[data-bs-toggle="list"]`;class Ks extends W{constructor(t){super(t),this._parent=this._element.closest('.list-group, .nav, [role="tablist"]'),this._parent&&(this._setInitialAttributes(this._parent,this._getChildren()),N.on(this._element,Ss,(t=>this._keydown(t))))}static get NAME(){return"tab"}show(){const t=this._element;if(this._elemIsActive(t))return;const e=this._getActiveElem(),i=e?N.trigger(e,Cs,{relatedTarget:t}):null;N.trigger(t,xs,{relatedTarget:e}).defaultPrevented||i&&i.defaultPrevented||(this._deactivate(e,t),this._activate(t,e))}_activate(t,e){t&&(t.classList.add(Fs),this._activate(z.getElementFromSelector(t)),this._queueCallback((()=>{"tab"===t.getAttribute("role")?(t.removeAttribute("tabindex"),t.setAttribute("aria-selected",!0),this._toggleDropDown(t,!0),N.trigger(t,ks,{relatedTarget:e})):t.classList.add(Ws)}),t,t.classList.contains(Hs)))}_deactivate(t,e){t&&(t.classList.remove(Fs),t.blur(),this._deactivate(z.getElementFromSelector(t)),this._queueCallback((()=>{"tab"===t.getAttribute("role")?(t.setAttribute("aria-selected",!1),t.setAttribute("tabindex","-1"),this._toggleDropDown(t,!1),N.trigger(t,Os,{relatedTarget:e})):t.classList.remove(Ws)}),t,t.classList.contains(Hs)))}_keydown(t){if(![$s,Is,Ns,Ps,js,Ms].includes(t.key))return;t.stopPropagation(),t.preventDefault();const e=this._getChildren().filter((t=>!l(t)));let i;if([js,Ms].includes(t.key))i=e[t.key===js?0:e.length-1];else{const n=[Is,Ps].includes(t.key);i=b(e,t.target,n,!0)}i&&(i.focus({preventScroll:!0}),Ks.getOrCreateInstance(i).show())}_getChildren(){return z.find(qs,this._parent)}_getActiveElem(){return this._getChildren().find((t=>this._elemIsActive(t)))||null}_setInitialAttributes(t,e){this._setAttributeIfNotExists(t,"role","tablist");for(const t of e)this._setInitialAttributesOnChild(t)}_setInitialAttributesOnChild(t){t=this._getInnerElement(t);const e=this._elemIsActive(t),i=this._getOuterElement(t);t.setAttribute("aria-selected",e),i!==t&&this._setAttributeIfNotExists(i,"role","presentation"),e||t.setAttribute("tabindex","-1"),this._setAttributeIfNotExists(t,"role","tab"),this._setInitialAttributesOnTargetPanel(t)}_setInitialAttributesOnTargetPanel(t){const e=z.getElementFromSelector(t);e&&(this._setAttributeIfNotExists(e,"role","tabpanel"),t.id&&this._setAttributeIfNotExists(e,"aria-labelledby",`${t.id}`))}_toggleDropDown(t,e){const i=this._getOuterElement(t);if(!i.classList.contains("dropdown"))return;const n=(t,n)=>{const s=z.findOne(t,i);s&&s.classList.toggle(n,e)};n(Bs,Fs),n(".dropdown-menu",Ws),i.setAttribute("aria-expanded",e)}_setAttributeIfNotExists(t,e,i){t.hasAttribute(e)||t.setAttribute(e,i)}_elemIsActive(t){return t.classList.contains(Fs)}_getInnerElement(t){return t.matches(qs)?t:z.findOne(qs,t)}_getOuterElement(t){return t.closest(".nav-item, .list-group-item")||t}static jQueryInterface(t){return this.each((function(){const e=Ks.getOrCreateInstance(this);if("string"==typeof t){if(void 0===e[t]||t.startsWith("_")||"constructor"===t)throw new TypeError(`No method named "${t}"`);e[t]()}}))}}N.on(document,Ls,Rs,(function(t){["A","AREA"].includes(this.tagName)&&t.preventDefault(),l(this)||Ks.getOrCreateInstance(this).show()})),N.on(window,Ds,(()=>{for(const t of z.find(Vs))Ks.getOrCreateInstance(t)})),m(Ks);const Qs=".bs.toast",Xs=`mouseover${Qs}`,Ys=`mouseout${Qs}`,Us=`focusin${Qs}`,Gs=`focusout${Qs}`,Js=`hide${Qs}`,Zs=`hidden${Qs}`,to=`show${Qs}`,eo=`shown${Qs}`,io="hide",no="show",so="showing",oo={animation:"boolean",autohide:"boolean",delay:"number"},ro={animation:!0,autohide:!0,delay:5e3};class ao extends W{constructor(t,e){super(t,e),this._timeout=null,this._hasMouseInteraction=!1,this._hasKeyboardInteraction=!1,this._setListeners()}static get Default(){return ro}static get DefaultType(){return oo}static get NAME(){return"toast"}show(){N.trigger(this._element,to).defaultPrevented||(this._clearTimeout(),this._config.animation&&this._element.classList.add("fade"),this._element.classList.remove(io),d(this._element),this._element.classList.add(no,so),this._queueCallback((()=>{this._element.classList.remove(so),N.trigger(this._element,eo),this._maybeScheduleHide()}),this._element,this._config.animation))}hide(){this.isShown()&&(N.trigger(this._element,Js).defaultPrevented||(this._element.classList.add(so),this._queueCallback((()=>{this._element.classList.add(io),this._element.classList.remove(so,no),N.trigger(this._element,Zs)}),this._element,this._config.animation)))}dispose(){this._clearTimeout(),this.isShown()&&this._element.classList.remove(no),super.dispose()}isShown(){return this._element.classList.contains(no)}_maybeScheduleHide(){this._config.autohide&&(this._hasMouseInteraction||this._hasKeyboardInteraction||(this._timeout=setTimeout((()=>{this.hide()}),this._config.delay)))}_onInteraction(t,e){switch(t.type){case"mouseover":case"mouseout":this._hasMouseInteraction=e;break;case"focusin":case"focusout":this._hasKeyboardInteraction=e}if(e)return void this._clearTimeout();const i=t.relatedTarget;this._element===i||this._element.contains(i)||this._maybeScheduleHide()}_setListeners(){N.on(this._element,Xs,(t=>this._onInteraction(t,!0))),N.on(this._element,Ys,(t=>this._onInteraction(t,!1))),N.on(this._element,Us,(t=>this._onInteraction(t,!0))),N.on(this._element,Gs,(t=>this._onInteraction(t,!1)))}_clearTimeout(){clearTimeout(this._timeout),this._timeout=null}static jQueryInterface(t){return this.each((function(){const e=ao.getOrCreateInstance(this,t);if("string"==typeof t){if(void 0===e[t])throw new TypeError(`No method named "${t}"`);e[t](this)}}))}}return R(ao),m(ao),{Alert:Q,Button:Y,Carousel:xt,Collapse:Bt,Dropdown:qi,Modal:On,Offcanvas:qn,Popover:us,ScrollSpy:Es,Tab:Ks,Toast:ao,Tooltip:cs}})); +//# sourceMappingURL=bootstrap.bundle.min.js.map \ No newline at end of file diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs new file mode 100644 index 0000000..e5a1714 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/BootTests.cs @@ -0,0 +1,193 @@ +using System.Net; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Boots the REAL Program.cs and asks the questions no in-process test can: does every +/// endpoint answer without a login, and does a bad registry stop the host rather than produce a +/// dashboard full of nothing. +/// +/// +/// The registry is supplied through environment variables rather than +/// ConfigureAppConfiguration. ConfigPreflight runs against +/// builder.Configuration in the top-level statements, before Build() — which is when +/// the factory's configuration callbacks are applied — so a callback-supplied registry would arrive +/// too late for the very check these tests are about. Environment variables are read by +/// CreateBuilder itself, so they are in place from the first line. +/// +/// +/// Environment variables are process-global, so every test here lives in one class: xunit runs a +/// class's tests sequentially, which is what keeps two hosts from fighting over the same keys. +/// +/// +public sealed class BootTests : IDisposable +{ + private readonly List _keys = []; + + private void Set(string key, string? value) + { + _keys.Add(key); + Environment.SetEnvironmentVariable(key, value); + } + + /// Restores the environment so a later-running test class sees a clean process. + public void Dispose() + { + foreach (var key in _keys) + Environment.SetEnvironmentVariable(key, null); + } + + /// + /// A registry with one instance pointing at a port nothing listens on. The dashboard must boot + /// and serve regardless of whether anything it watches is up — that is the entire premise of + /// rendering from cache. + /// + private void SetValidRegistry() + { + Set("ASPNETCORE_ENVIRONMENT", "Testing"); + Set("Overview__PollIntervalSeconds", "10"); + Set("Overview__TimeoutSeconds", "1"); + Set("Overview__StaleAfterSeconds", "45"); + Set("Overview__Applications__0__Name", "TestApp"); + Set("Overview__Applications__0__ManagementLabel", "UI"); + Set("Overview__Applications__0__Instances__0__Name", "node-a"); + Set("Overview__Applications__0__Instances__0__BaseUrl", "http://127.0.0.1:1"); + Set("Overview__Applications__0__Instances__0__HasActiveRole", "false"); + } + + private static WebApplicationFactory Factory() => new(); + + [Fact] + public async Task Page_IsServedAnonymously() + { + SetValidRegistry(); + using var factory = Factory(); + + // AllowAutoRedirect off so an auth redirect would surface as a 302 here rather than being + // silently followed to a login page that then returns 200. + using var client = factory.CreateClient(new WebApplicationFactoryClientOptions + { + AllowAutoRedirect = false, + }); + + using var response = await client.GetAsync("/"); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Contains("SCADA Overview", await response.Content.ReadAsStringAsync(), StringComparison.Ordinal); + } + + [Fact] + public async Task Page_RendersEvenThoughNothingItWatchesIsReachable() + { + // Cache-first, stated as a test: the registry above points at a dead port, and the page + // must still paint its full layout instead of waiting on a sweep. + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + var html = await client.GetStringAsync("/"); + + Assert.Contains("data-testid=\"instance-card\"", html, StringComparison.Ordinal); + Assert.Contains("node-a", html, StringComparison.Ordinal); + } + + [Theory] + [InlineData("/health/ready")] + [InlineData("/health/active")] + [InlineData("/healthz")] + [InlineData("/metrics")] + public async Task HealthAndMetrics_AreServedAnonymously(string path) + { + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(new WebApplicationFactoryClientOptions + { + AllowAutoRedirect = false, + }); + + using var response = await client.GetAsync(path); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + } + + [Fact] + public async Task Ready_ReportsTheDashboardsOwnChecks() + { + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + var body = await client.GetStringAsync("/health/ready"); + + Assert.Contains(ObservabilityServiceCollectionExtensions.RegistryLoadedCheckName, body, StringComparison.Ordinal); + Assert.Contains(ObservabilityServiceCollectionExtensions.PollCycleCheckName, body, StringComparison.Ordinal); + } + + [Fact] + public async Task Metrics_ExportsTheAppsOwnMeter() + { + // The Meters allowlist in AddZbTelemetry is silent when it is wrong: a missing name drops + // the app's whole instrument surface from /metrics with no warning anywhere. Only an + // end-to-end scrape catches that. + SetValidRegistry(); + using var factory = Factory(); + using var client = factory.CreateClient(); + + // Force a sweep so the observable gauge has a probed instance to report. The instance is + // unreachable, which is a perfectly good status to publish. + var poller = factory.Services.GetServices().OfType().Single(); + await poller.SweepAsync(); + + var body = await client.GetStringAsync("/metrics"); + + Assert.Contains("overview_instance_status", body, StringComparison.Ordinal); + Assert.Contains("node=\"node-a\"", body, StringComparison.Ordinal); + } + + [Fact] + public async Task MissingRegistry_FailsToBootAndNamesTheKey() + { + // appsettings.json ships an EMPTY Applications list precisely so this is the outcome when a + // deployment forgets its registry — an empty dashboard that boots would look like a healthy + // fleet of nothing. + Set("ASPNETCORE_ENVIRONMENT", "Testing"); + + using var factory = Factory(); + + var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/")); + + Assert.NotNull(error); + Assert.Contains("Overview:Applications:0:Name", Flatten(error), StringComparison.Ordinal); + } + + [Fact] + public async Task StaleWindowInsideThePollInterval_FailsToBootWithTheValidatorMessage() + { + // Past preflight and into the validator: a staleness window shorter than the poll interval + // would mark every instance Stale between two perfectly healthy polls. + SetValidRegistry(); + Set("Overview__StaleAfterSeconds", "5"); + + using var factory = Factory(); + + var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/")); + + Assert.NotNull(error); + Assert.Contains("StaleAfterSeconds", Flatten(error), StringComparison.Ordinal); + } + + private static string Flatten(Exception exception) + { + var text = new System.Text.StringBuilder(); + + for (var current = exception; current is not null; current = current.InnerException) + text.AppendLine(current.Message); + + return text.ToString(); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/InstanceCardTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/InstanceCardTests.cs new file mode 100644 index 0000000..fb3d328 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/InstanceCardTests.cs @@ -0,0 +1,249 @@ +using Bunit; +using ZB.MOM.WW.Overview.Components.Widgets; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// The instance card is where a reading becomes a decision, so the tests pin what an operator can +/// actually distinguish at a glance: which of the four states is shown, whether it is still +/// current, and whether the raw evidence behind it is visible. +/// +public class InstanceCardTests : TestContext +{ + private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero); + + private static InstanceSnapshot Card( + InstanceStatus? status = InstanceStatus.Up, + ActiveState active = ActiveState.NotApplicable, + DateTimeOffset? lastPolled = null, + DateTimeOffset? lastReachable = null, + TimeSpan? staleAfter = null, + string? error = null, + ZbHealthReport? report = null, + string? managementUrl = null) => + new() + { + Application = "ScadaBridge", + Instance = "site-a-a", + Group = "site-a", + BaseUrl = "http://site-a-a:8084", + ManagementUrl = managementUrl, + ManagementLabel = "AdminUI", + HasActiveRole = active != ActiveState.NotApplicable, + Status = status, + Active = active, + LastPolledUtc = lastPolled ?? Now, + LastReachableUtc = lastReachable ?? Now, + Latency = TimeSpan.FromMilliseconds(18), + Error = error, + Report = report, + StaleAfter = staleAfter ?? TimeSpan.FromSeconds(45), + }; + + private IRenderedComponent Render(InstanceSnapshot instance, bool isLeader = false) => + RenderComponent(p => p + .Add(c => c.Instance, instance) + .Add(c => c.Now, Now) + .Add(c => c.IsLeader, isLeader)); + + [Theory] + [InlineData(InstanceStatus.Up, "up", "Up")] + [InlineData(InstanceStatus.Degraded, "degraded", "Degraded")] + [InlineData(InstanceStatus.Down, "down", "Down")] + [InlineData(InstanceStatus.Unreachable, "unreachable", "Unreachable")] + public void Card_CarriesItsStatusAsBothClassAndLabel(InstanceStatus status, string cssClass, string label) + { + var card = Render(Card(status)); + + Assert.Contains(cssClass, card.Find("article").ClassList, StringComparer.Ordinal); + Assert.Contains(label, card.Markup, StringComparison.Ordinal); + } + + [Fact] + public void UnreachableAndDown_AreVisuallyDistinct_NotJustDifferentWords() + { + // The CSS gives Unreachable a dashed border precisely because both are red. If the class + // did not differ, the two would be indistinguishable on a wall display. + var down = Render(Card(InstanceStatus.Down)).Find("article").ClassList; + var unreachable = Render(Card(InstanceStatus.Unreachable)).Find("article").ClassList; + + Assert.NotEqual(down, unreachable); + } + + [Fact] + public void NeverPolled_RendersPending_NotHealthy() + { + var card = Render(Card(status: null, lastPolled: null)); + + Assert.Contains("pending", card.Find("article").ClassList, StringComparer.Ordinal); + Assert.Contains("Pending", card.Markup, StringComparison.Ordinal); + Assert.DoesNotContain(">Up<", card.Markup, StringComparison.Ordinal); + } + + [Fact] + public void StaleCard_IsMarkedStaleButKeepsItsLastKnownStatusClass() + { + // Both facts matter: the reading was Up, and it is no longer current. Dropping either one + // loses information the operator needs. + var card = Render(Card(lastPolled: Now.AddMinutes(-5))); + + var classes = card.Find("article").ClassList; + Assert.Contains("stale", classes, StringComparer.Ordinal); + Assert.Contains("up", classes, StringComparer.Ordinal); + Assert.Contains("Stale", card.Markup, StringComparison.Ordinal); + } + + [Fact] + public void StaleCard_CountsFromLastContact_NotFromTheLastPollAttempt() + { + // The poller kept trying for ten minutes and kept failing. "Stale since 5 s ago" would + // suggest the instance was fine moments ago; it has actually been gone for ten minutes. + var card = Render(Card( + status: InstanceStatus.Unreachable, + lastPolled: Now.AddSeconds(-5), + lastReachable: Now.AddMinutes(-10), + staleAfter: TimeSpan.FromSeconds(1))); + + Assert.Contains("Stale since", card.Markup, StringComparison.Ordinal); + Assert.Contains("10 min ago", card.Markup, StringComparison.Ordinal); + } + + [Theory] + [InlineData(ActiveState.Active, "Active")] + [InlineData(ActiveState.Standby, "Standby")] + [InlineData(ActiveState.Unknown, "Role ?")] + public void ActiveState_IsBadged(ActiveState state, string expected) + { + Assert.Contains(expected, Render(Card(active: state)).Markup, StringComparison.Ordinal); + } + + [Fact] + public void NoActiveRole_ShowsNoActivityBadgeAtAll() + { + // A single-instance gateway has no active/standby concept; a badge would invent one. + var markup = Render(Card(active: ActiveState.NotApplicable)).Markup; + + Assert.DoesNotContain("Standby", markup, StringComparison.Ordinal); + Assert.DoesNotContain("Role ?", markup, StringComparison.Ordinal); + Assert.DoesNotContain("· active", markup, StringComparison.Ordinal); + } + + [Fact] + public void LeaderBadge_IsShownOnlyWhenTheCardIsTheLeader() + { + Assert.Contains("Leader", Render(Card(), isLeader: true).Markup, StringComparison.Ordinal); + Assert.DoesNotContain("Leader", Render(Card(), isLeader: false).Markup, StringComparison.Ordinal); + } + + [Fact] + public void Checks_AreListedWithAHealthyCount() + { + var card = Render(Card(report: Report(("localdb", "Healthy", null), ("akka-cluster", "Unhealthy", null)))); + + Assert.Contains("1 / 2 healthy", card.Markup, StringComparison.Ordinal); + Assert.Equal(2, card.FindAll(".check-row").Count); + } + + [Fact] + public void ClusterData_IsRenderedAsRawEvidence() + { + var card = Render(Card(report: Report(("akka-cluster", "Healthy", "akka.tcp://scadabridge@site-a-a:8082")))); + + var data = card.Find(".check-data"); + Assert.Contains("akka.tcp://scadabridge@site-a-a:8082", data.TextContent, StringComparison.Ordinal); + Assert.Contains("leader", data.TextContent, StringComparison.Ordinal); + } + + [Fact] + public void ClusterData_FromAnInstanceWithoutIt_IsOmittedEntirely() + { + // Every pre-0.2.0 node, and every check that publishes no data. The card must simply not + // show the line rather than showing an empty one. + var card = Render(Card(report: Report(("localdb", "Healthy", null)))); + + Assert.Empty(card.FindAll(".check-data")); + } + + [Fact] + public void ClusterData_IsHtmlEscaped() + { + // The value is remote input from another application. Operationally trusted, but a health + // check that ever emits markup must not be able to inject it into this page. + var card = Render(Card(report: Report(("akka-cluster", "Healthy", "")))); + + var data = card.Find(".check-data"); + Assert.Empty(data.QuerySelectorAll("img")); + Assert.Contains("", data.TextContent, StringComparison.Ordinal); + } + + [Fact] + public void ManagementLink_OpensInANewTabWithNoopener() + { + var link = Render(Card(managementUrl: "http://site-a-a:9000/")).Find(".panel-foot a"); + + Assert.Equal("http://site-a-a:9000/", link.GetAttribute("href")); + Assert.Equal("_blank", link.GetAttribute("target")); + Assert.Equal("noopener", link.GetAttribute("rel")); + Assert.Contains("AdminUI", link.TextContent, StringComparison.Ordinal); + } + + [Fact] + public void NoManagementUrl_RendersNoDeadLink() + { + var card = Render(Card(managementUrl: null)); + + Assert.Empty(card.FindAll(".panel-foot a")); + Assert.Contains("no management link", card.Markup, StringComparison.Ordinal); + } + + [Theory] + [InlineData(InstanceStatus.Up, ActiveState.Active, "ready 200 · active 200")] + [InlineData(InstanceStatus.Up, ActiveState.Standby, "ready 200 · active 503")] + [InlineData(InstanceStatus.Down, ActiveState.NotApplicable, "ready 503")] + [InlineData(InstanceStatus.Unreachable, ActiveState.NotApplicable, "ready fail")] + public void RawSignalLine_ShowsWhatWasActuallyReceived( + InstanceStatus status, ActiveState active, string expected) + { + // The footer is the escape hatch from every derived label above it: when the dashboard and + // a curl disagree, this line is what reconciles them. + Assert.Contains( + expected, + Render(Card(status, active)).Find("[data-testid=raw-signals]").TextContent, + StringComparison.Ordinal); + } + + [Fact] + public void ProbeError_IsSurfacedOnTheCard() + { + var card = Render(Card(InstanceStatus.Unreachable, error: "Connection refused")); + + Assert.Contains("Connection refused", card.Markup, StringComparison.Ordinal); + } + + [Fact] + public void Host_IsShownWithoutTheScheme() + { + Assert.Contains("site-a-a:8084", Render(Card()).Find(".inst-host").TextContent, StringComparison.Ordinal); + } + + [Theory] + [InlineData(4, "4 s ago")] + [InlineData(90, "2 min ago")] + [InlineData(7200, "2 h ago")] + [InlineData(172800, "2 d ago")] + public void Relative_ScalesTheUnit(int seconds, string expected) + { + Assert.Equal(expected, InstanceCard.Relative(TimeSpan.FromSeconds(seconds))); + } + + [Fact] + public void Relative_NegativeAge_ReadsAsNow_NotAsTheFuture() + { + // Clock skew between this host and a probed one is routine; "-3 s ago" is nonsense. + Assert.Equal("0 s ago", InstanceCard.Relative(TimeSpan.FromSeconds(-3))); + } + + private static ZbHealthReport Report(params (string Name, string Status, string? Leader)[] entries) => + System.Text.Json.JsonSerializer.Deserialize(HealthBody.Ready("Healthy", entries))!; +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/LeaderResolverTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/LeaderResolverTests.cs new file mode 100644 index 0000000..8704dc5 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/LeaderResolverTests.cs @@ -0,0 +1,283 @@ +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Leader aggregation and the split-brain flag. The disagreement rule is the one output an operator +/// would be woken by, so both directions are pinned: it must fire when two members genuinely claim +/// different leaders, and it must NOT fire for the ordinary cases that superficially resemble that +/// — a node that stopped answering, a node running a pre-0.2.0 Health package. +/// +public class LeaderResolverTests +{ + private const string LeaderA = "akka.tcp://scadabridge@site-a-a:8082"; + private const string LeaderB = "akka.tcp://scadabridge@site-a-b:8082"; + + [Fact] + public void Resolve_AgreeingPair_NamesTheLeaderInstance() + { + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderA), + ]); + + var group = Assert.Single(groups); + Assert.Equal("site-a", group.Name); + Assert.Equal("site-a-a", group.LeaderInstance); + Assert.Equal(LeaderA, group.LeaderAddress); + Assert.False(group.Disagreement); + } + + [Fact] + public void Resolve_MembersClaimDifferentLeaders_FlagsDisagreement() + { + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB), + ]); + + var group = Assert.Single(groups); + Assert.True(group.Disagreement); + Assert.Equal([LeaderA, LeaderB], group.ReportedLeaders); + } + + [Theory] + [InlineData(InstanceStatus.Down)] + [InlineData(InstanceStatus.Unreachable)] + public void Resolve_FailedMemberDoesNotVote_SoAFailoverIsNotMisreadAsSplitBrain(InstanceStatus failed) + { + // Mid-failover the departed node's last body still names the old leader. Counting it would + // raise a split-brain warning during the most normal event in a redundant pair's life. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: failed, leader: LeaderA), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB), + ]); + + var group = Assert.Single(groups); + Assert.False(group.Disagreement); + Assert.Equal("site-a-b", group.LeaderInstance); + } + + [Fact] + public void Resolve_DegradedMemberStillVotes() + { + // Degraded means answering with a full cluster view — its opinion on the leader is current. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: InstanceStatus.Degraded, leader: LeaderA), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB), + ]); + + Assert.True(Assert.Single(groups).Disagreement); + } + + [Fact] + public void Resolve_NoMemberPublishesLeaderData_YieldsNoLeaderAndNoWarning() + { + // The whole-fleet-on-Health-0.1.0 case: no data key anywhere. It must degrade to "leader + // unknown", never to a false split-brain warning. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084"), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"), + ]); + + var group = Assert.Single(groups); + Assert.Null(group.LeaderAddress); + Assert.Null(group.LeaderInstance); + Assert.False(group.Disagreement); + Assert.Empty(group.ReportedLeaders); + } + + [Fact] + public void Resolve_OneMemberOnOldHealthPackage_UsesTheOneThatDoesReport() + { + // A partly-bumped fleet is the expected state during rollout, and one silent member is not + // a disagreement. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA), + Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"), + ]); + + var group = Assert.Single(groups); + Assert.Equal("site-a-a", group.LeaderInstance); + Assert.False(group.Disagreement); + } + + [Fact] + public void Resolve_UnresolvableAddress_KeepsTheRawAddress() + { + // A leader on a host that is not in the registry (a node nobody added) still tells the + // operator something useful — showing nothing would hide the discovery. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: "akka.tcp://scadabridge@ghost:8082"), + ]); + + var group = Assert.Single(groups); + Assert.Null(group.LeaderInstance); + Assert.Equal("akka.tcp://scadabridge@ghost:8082", group.LeaderAddress); + } + + [Fact] + public void Resolve_MatchesOnHostOnly_BecauseAkkaAndHttpPortsDiffer() + { + // The registry knows the HTTP port (8084); the leader address carries the Akka remoting + // port (8082). Matching on authority instead of host would never resolve anything. + var name = LeaderResolver.ResolveToInstanceName( + LeaderA, + [Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084")]); + + Assert.Equal("site-a-a", name); + } + + [Fact] + public void Resolve_GroupWithoutActiveRole_IsNotAGroupAtAll() + { + // Instances grouped only for layout have no leader; a chip there would invent a concept. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("gw-1", "gateways", "http://gw-1:5120", hasActiveRole: false), + Snapshots.Instance("gw-2", "gateways", "http://gw-2:5120", hasActiveRole: false), + ]); + + Assert.Empty(groups); + } + + [Fact] + public void Resolve_UngroupedInstances_ProduceNoGroups() + { + Assert.Empty(LeaderResolver.Resolve([Snapshots.Instance("solo")])); + } + + [Fact] + public void Resolve_SeparateGroups_AreScopedIndependently() + { + // Two site pairs, each with its own leader: per-Cluster election is the family's actual + // topology, so a leader from one group must never be attributed to another. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA), + Snapshots.Instance("site-b-a", "site-b", "http://site-b-a:8084", leader: "akka.tcp://scadabridge@site-b-a:8082"), + ]); + + Assert.Equal(2, groups.Count); + Assert.Equal("site-a-a", groups[0].LeaderInstance); + Assert.Equal("site-b-a", groups[1].LeaderInstance); + Assert.All(groups, g => Assert.False(g.Disagreement)); + } + + [Theory] + [InlineData("akka.tcp://scadabridge@site-a-a:8082", "site-a-a")] + [InlineData("akka.tcp://otopcua@10.100.0.35:4053", "10.100.0.35")] + [InlineData("akka.tcp://sys@host", "host")] + [InlineData("not-an-address", null)] + [InlineData("", null)] + [InlineData(null, null)] + public void HostOf_ParsesAkkaAddresses(string? address, string? expected) + { + Assert.Equal(expected, LeaderResolver.HostOf(address)); + } + + [Theory] + [InlineData("akka-cluster")] // ScadaBridge + [InlineData("akka")] // OtOpcUa + [InlineData("cluster-view")] // any future app + public void Leader_IsFoundWhateverTheCheckIsCalled(string checkName) + { + const string NodeALeader = "akka.tcp://otopcua@node-a:4053"; + + // The two apps register the SAME shared AkkaClusterHealthCheck under different names, so a + // resolver keyed on one name renders no leader chip at all for the other — silently, since + // a group with no votes is a legitimate state. That is exactly how this shipped past + // review and unit tests and was only caught on a live rig. + var groups = LeaderResolver.Resolve( + [ + Snapshots.Instance("node-a", "pair", "http://node-a:8080", leader: NodeALeader, clusterCheckName: checkName), + Snapshots.Instance("node-b", "pair", "http://node-b:8080", leader: NodeALeader, clusterCheckName: checkName), + ]); + + var group = Assert.Single(groups); + Assert.Equal("node-a", group.LeaderInstance); + Assert.False(group.Disagreement); + } + + [Fact] + public void Leader_IgnoresChecksThatPublishDataWithoutALeader() + { + // ScadaBridge's site nodes publish a localdb check whose data carries replication counters + // and no leader. Scanning entries must pick the one that actually answers the question, + // not merely the first one carrying a data object. + var body = HealthBody.Ready( + "Healthy", + ("localdb", "Healthy", null), + ("akka", "Healthy", "akka.tcp://otopcua@node-a:4053")); + var report = System.Text.Json.JsonSerializer.Deserialize(body)!; + var instance = Snapshots.Instance("node-a", "pair", "http://node-a:8080") with { Report = report }; + + Assert.Equal("akka.tcp://otopcua@node-a:4053", LeaderResolver.LeaderReportedBy(instance)); + } + + [Fact] + public void ActiveNode_WinsOverLeader_WhenTheyDisagree() + { + // The two are DIFFERENT NODES after any restart: leader is the lowest-addressed Up member, + // the active node is the oldest. Observed live on the rebuilt OtOpcUa rig — leader central-1, + // active central-2 — so ranking leader first would put the Active chip on the standby, which + // is the same class of error as lmxopcua#494 rendered in the UI instead of in routing. + var body = HealthBody.WithData( + "Healthy", + ("akka", "Healthy", new Dictionary + { + ["leader"] = "akka.tcp://otopcua@central-1:4053", + }), + ("cluster-primary", "Healthy", new Dictionary + { + ["activeNode"] = "akka.tcp://otopcua@central-2:4053", + })); + var report = System.Text.Json.JsonSerializer.Deserialize(body)!; + var instance = Snapshots.Instance("central-1", "MAIN", "http://central-1:8080") with { Report = report }; + + Assert.Equal("akka.tcp://otopcua@central-2:4053", LeaderResolver.LeaderReportedBy(instance)); + } + + [Fact] + public void ActiveNode_ReportedByAStandby_NamesTheActiveNode_NotItself() + { + // Every member publishes who IS active, so a standby votes for the same address the active + // node does. That is what lets the group resolve with one unanimous chip instead of a + // manufactured disagreement, and it is why the standby's payload alone is enough. + var body = HealthBody.WithData( + "Unhealthy", + ("cluster-primary", "Unhealthy", new Dictionary + { + ["activeNode"] = "akka.tcp://otopcua@site-a-1:4053", + ["selfAddress"] = "akka.tcp://otopcua@site-a-2:4053", + })); + var report = System.Text.Json.JsonSerializer.Deserialize(body)!; + var instance = Snapshots.Instance("site-a-2", "SITE-A", "http://site-a-2:8080") with { Report = report }; + + Assert.Equal("akka.tcp://otopcua@site-a-1:4053", LeaderResolver.LeaderReportedBy(instance)); + } + + [Fact] + public void Leader_IsStillUsed_WhenNoActiveNodeIsPublished() + { + // An instance on Health < 0.3.0, or one whose active tier was not probed, publishes only + // `leader`. Showing it beats showing nothing, and it keeps the pre-0.3.0 behaviour intact. + var body = HealthBody.WithData( + "Healthy", + ("akka-cluster", "Healthy", new Dictionary + { + ["leader"] = "akka.tcp://scadabridge@central-a:4053", + })); + var report = System.Text.Json.JsonSerializer.Deserialize(body)!; + var instance = Snapshots.Instance("central-a", "central", "http://central-a:8080") with { Report = report }; + + Assert.Equal("akka.tcp://scadabridge@central-a:4053", LeaderResolver.LeaderReportedBy(instance)); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs new file mode 100644 index 0000000..94919ef --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ObservabilityTests.cs @@ -0,0 +1,309 @@ +using System.Diagnostics.Metrics; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// The dashboard watching itself: the gauge it exports about the fleet, and the two checks that +/// decide whether its own /health/ready is telling the truth. +/// +public class ObservabilityTests +{ + private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero); + + private static OverviewOptions Registry( + int pollIntervalSeconds = 10, + int timeoutSeconds = 3, + params ApplicationEntry[] applications) => + new() + { + PollIntervalSeconds = pollIntervalSeconds, + TimeoutSeconds = timeoutSeconds, + StaleAfterSeconds = 45, + Applications = applications.Length > 0 + ? applications + : [new ApplicationEntry + { + Name = "App", + Instances = [new InstanceEntry { Name = "a", BaseUrl = "http://a:8080" }], + }], + }; + + private static HealthCheckContext Context() => new(); + + // ---------------------------------------------------------------- registry-loaded + + [Fact] + public async Task RegistryLoaded_ReportsTheCountsItBound() + { + var options = Registry(applications: + [ + new ApplicationEntry + { + Name = "ScadaBridge", + Instances = + [ + new InstanceEntry { Name = "central-a", BaseUrl = "http://central-a:9000" }, + new InstanceEntry { Name = "central-b", BaseUrl = "http://central-b:9100" }, + ], + }, + new ApplicationEntry + { + Name = "OtOpcUa", + Instances = [new InstanceEntry { Name = "admin", BaseUrl = "http://admin:9200" }], + }, + ]); + + var result = await new RegistryLoadedHealthCheck(Options.Create(options)) + .CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Healthy, result.Status); + Assert.Equal(2, result.Data["applications"]); + Assert.Equal(3, result.Data["instances"]); + } + + [Fact] + public async Task RegistryLoaded_EmptyRegistry_IsUnhealthy() + { + // Unreachable through the real startup path — preflight and the validator both refuse it — + // but the check must still be able to say so, or it is only ever decorative. + var options = new OverviewOptions { Applications = [] }; + + var result = await new RegistryLoadedHealthCheck(Options.Create(options)) + .CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Equal(0, result.Data["instances"]); + } + + // ---------------------------------------------------------------- grace window + + [Fact] + public void Grace_IsTwiceTheFastestIntervalPlusTheSlowestTimeout() + { + var options = Registry(pollIntervalSeconds: 10, timeoutSeconds: 3); + + Assert.Equal(TimeSpan.FromSeconds(26), PollCycleHealthCheck.GraceFor(options)); + } + + [Fact] + public void Grace_HonoursPerInstanceOverrides() + { + // The sweep loop ticks at the FASTEST configured interval, and the slowest timeout bounds + // how long any one sweep can take — so the window has to be derived from both extremes, + // not from the global defaults. + var options = Registry(pollIntervalSeconds: 30, timeoutSeconds: 3, applications: + [ + new ApplicationEntry + { + Name = "App", + Instances = + [ + new InstanceEntry { Name = "fast", BaseUrl = "http://fast:8080", PollIntervalSeconds = 5 }, + new InstanceEntry { Name = "slow", BaseUrl = "http://slow:8080", TimeoutSeconds = 20 }, + ], + }, + ]); + + Assert.Equal(TimeSpan.FromSeconds(50), PollCycleHealthCheck.GraceFor(options)); + } + + // ---------------------------------------------------------------- last-poll-cycle-completed + + [Fact] + public async Task PollCycle_NoCycleYetButStillStarting_IsDegradedNotUnhealthy() + { + // Every deployment passes through this state. Failing readiness here would make the + // container look broken for the first seconds of its life, every single restart. + var time = new FakeTimeProvider(Start); + var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time); + + time.Advance(TimeSpan.FromSeconds(5)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Degraded, result.Status); + } + + [Fact] + public async Task PollCycle_NeverSwept_EventuallyGoesUnhealthy() + { + // A poller that never started must not hide behind "still starting" forever. + var time = new FakeTimeProvider(Start); + var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time); + + time.Advance(TimeSpan.FromMinutes(5)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Contains("since startup", result.Description, StringComparison.Ordinal); + } + + [Fact] + public async Task PollCycle_RecentSweep_IsHealthyAndPublishesItsAge() + { + var time = new FakeTimeProvider(Start); + var heartbeat = new PollCycleHeartbeat(time); + var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time); + + heartbeat.RecordCycleCompleted(Start); + time.Advance(TimeSpan.FromSeconds(4)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Healthy, result.Status); + Assert.Equal(4d, result.Data["ageSeconds"]); + Assert.Equal(26d, result.Data["graceSeconds"]); + } + + [Fact] + public async Task PollCycle_StoppedSweeping_IsUnhealthy() + { + // The failure this check exists for: Kestrel is fine, the page still renders, and every + // card on it is quietly frozen. + var time = new FakeTimeProvider(Start); + var heartbeat = new PollCycleHeartbeat(time); + var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time); + + heartbeat.RecordCycleCompleted(Start); + time.Advance(TimeSpan.FromMinutes(2)); + var result = await check.CheckHealthAsync(Context()); + + Assert.Equal(HealthStatus.Unhealthy, result.Status); + Assert.Contains("stale", result.Description, StringComparison.Ordinal); + } + + [Fact] + public void Heartbeat_ReportsNothingUntilACycleCompletes() + { + var heartbeat = new PollCycleHeartbeat(new FakeTimeProvider(Start)); + + Assert.Null(heartbeat.LastCompletedUtc); + Assert.Equal(Start, heartbeat.StartedUtc); + + heartbeat.RecordCycleCompleted(Start.AddSeconds(7)); + + Assert.Equal(Start.AddSeconds(7), heartbeat.LastCompletedUtc); + } + + // ---------------------------------------------------------------- the fleet gauge + + [Fact] + public void Gauge_ReportsOneMeasurementPerProbedInstance() + { + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf( + Snapshots.Instance("central-a", group: "central", status: InstanceStatus.Up), + Snapshots.Instance("central-b", group: "central", status: InstanceStatus.Degraded), + Snapshots.Instance("site-a-a", group: "site-a", status: InstanceStatus.Unreachable))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + var measured = CollectStatusGauge(factory); + + Assert.Equal(3, measured.Count); + Assert.Equal(0, measured.Single(m => m.Tags["node"] as string == "central-a").Value); + Assert.Equal(1, measured.Single(m => m.Tags["node"] as string == "central-b").Value); + Assert.Equal(3, measured.Single(m => m.Tags["node"] as string == "site-a-a").Value); + Assert.Equal("central", measured.First().Tags["group"]); + Assert.Equal("App", measured.First().Tags["application"]); + } + + [Fact] + public void Gauge_SkipsInstancesThatHaveNeverBeenProbed() + { + // The enum's zero ordinal is Up, so emitting anything for an unprobed instance would + // publish the single most reassuring answer about a node nobody has heard from. + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf( + Snapshots.Instance("probed", status: InstanceStatus.Down), + Snapshots.Instance("never-probed", status: null))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + var measured = CollectStatusGauge(factory); + + Assert.Equal("probed", Assert.Single(measured).Tags["node"]); + } + + [Fact] + public void Gauge_TracksTheStoreRatherThanASnapshotTakenAtConstruction() + { + // It is an OBSERVABLE gauge: read at scrape time from whatever the store currently holds, + // so the metric and the page can never disagree about what the dashboard believes. + var store = new OverviewSnapshotStore(); + store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Up))); + + using var factory = new DummyMeterFactory(); + using var metrics = new OverviewMetrics(factory, store); + + Assert.Equal(0, Assert.Single(CollectStatusGauge(factory)).Value); + + store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Down))); + + Assert.Equal(2, Assert.Single(CollectStatusGauge(factory)).Value); + } + + [Fact] + public async Task Sweep_StampsTheHeartbeatSoTheSelfCheckCanSeeIt() + { + // Ties the two halves together: without this the health check would be measuring a value + // nothing ever writes, and would report a healthy poller forever. + var time = new FakeTimeProvider(Start); + var store = new OverviewSnapshotStore(); + var heartbeat = new PollCycleHeartbeat(time); + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", System.Net.HttpStatusCode.OK, HealthBody.Ready("Healthy")); + + using var factory = new DummyMeterFactory(); + var poller = new OverviewPollerService( + Options.Create(Registry()), + new StubHttpClientFactory(handler), + store, + time, + new OverviewMetrics(factory, store), + heartbeat, + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + + Assert.Null(heartbeat.LastCompletedUtc); + + await poller.SweepAsync(); + + Assert.Equal(Start, heartbeat.LastCompletedUtc); + } + + private static OverviewSnapshot SnapshotOf(params InstanceSnapshot[] instances) => + new(Start, [new ApplicationSnapshot("App", instances, [])]); + + private static List<(int Value, Dictionary Tags)> CollectStatusGauge( + DummyMeterFactory factory) + { + var measured = new List<(int, Dictionary)>(); + + using var listener = new MeterListener + { + InstrumentPublished = (instrument, l) => + { + // Scoped to THIS test's factory: meter names are process-global, so matching on + // the name alone would also collect instruments from tests running in parallel. + if (ReferenceEquals(instrument.Meter.Scope, factory) && + instrument.Name == OverviewMetrics.InstanceStatusInstrument) + { + l.EnableMeasurementEvents(instrument); + } + }, + }; + + listener.SetMeasurementEventCallback((_, value, tags, _) => + measured.Add((value, tags.ToArray().ToDictionary(t => t.Key, t => t.Value, StringComparer.Ordinal)))); + + listener.Start(); + listener.RecordObservableInstruments(); + + return measured; + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewOptionsValidatorTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewOptionsValidatorTests.cs new file mode 100644 index 0000000..ae5aab8 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewOptionsValidatorTests.cs @@ -0,0 +1,226 @@ +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Registry validation. A malformed registry must fail the host at startup with a message naming +/// the offending key — the alternative is a dashboard that boots and quietly shows nothing, or one +/// that throws on its first poll where nobody is looking. +/// +public class OverviewOptionsValidatorTests +{ + private static readonly OverviewOptionsValidator Validator = new(); + + private static OverviewOptions Valid(Action? mutate = null) + { + var options = new OverviewOptions + { + PollIntervalSeconds = 10, + TimeoutSeconds = 3, + StaleAfterSeconds = 45, + Applications = + { + new ApplicationEntry + { + Name = "OtOpcUa", + ManagementLabel = "AdminUI", + Instances = + { + new InstanceEntry + { + Name = "central-1", + Group = "central", + BaseUrl = "http://otopcua-central-1:9000", + ManagementUrl = "http://otopcua-central-1:9000/", + HasActiveRole = true, + }, + }, + }, + }, + }; + + mutate?.Invoke(options); + return options; + } + + private static ValidateOptionsResult Run(OverviewOptions options) => Validator.Validate(null, options); + + private static void AssertFailsWith(OverviewOptions options, string expectedFragment) + { + var result = Run(options); + + Assert.True(result.Failed, "expected validation to fail"); + Assert.Contains( + result.Failures, + f => f.Contains(expectedFragment, StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void ValidRegistry_Succeeds() + { + // Positive control: without this, every "fails with X" assertion below could be passing + // because the fixture itself is malformed for some unrelated reason. + Assert.True(Run(Valid()).Succeeded); + } + + [Fact] + public void EmptyRegistry_IsRejected() + { + AssertFailsWith(Valid(o => o.Applications.Clear()), "Overview:Applications"); + } + + [Fact] + public void ApplicationWithNoInstances_IsRejected() + { + AssertFailsWith(Valid(o => o.Applications[0].Instances.Clear()), "Instances"); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + public void ApplicationWithoutName_IsRejected(string name) + { + AssertFailsWith(Valid(o => o.Applications[0].Name = name), "Applications:0:Name"); + } + + [Fact] + public void DuplicateApplicationNames_AreRejected() + { + AssertFailsWith( + Valid(o => o.Applications.Add(new ApplicationEntry + { + Name = "OtOpcUa", + Instances = { new InstanceEntry { Name = "other", BaseUrl = "http://other:9000" } }, + })), + "duplicated"); + } + + [Fact] + public void DuplicateInstanceNamesWithinAnApplication_AreRejected() + { + AssertFailsWith( + Valid(o => o.Applications[0].Instances.Add(new InstanceEntry + { + Name = "central-1", + BaseUrl = "http://otopcua-central-2:9000", + })), + "duplicated"); + } + + [Fact] + public void SameInstanceNameInDifferentApplications_IsAllowed() + { + // "central-1" under OtOpcUa and "central-1" under ScadaBridge are different machines with + // the same conventional name — uniqueness is per application, not global. + var options = Valid(o => o.Applications.Add(new ApplicationEntry + { + Name = "ScadaBridge", + Instances = { new InstanceEntry { Name = "central-1", BaseUrl = "http://sb-central-a:5000" } }, + })); + + Assert.True(Run(options).Succeeded); + } + + [Theory] + [InlineData("")] + [InlineData("otopcua-central-1:9000")] // no scheme + [InlineData("/health/ready")] // relative + [InlineData("ftp://otopcua-central-1")] // wrong scheme + public void NonAbsoluteHttpBaseUrl_IsRejected(string baseUrl) + { + AssertFailsWith(Valid(o => o.Applications[0].Instances[0].BaseUrl = baseUrl), "BaseUrl"); + } + + [Fact] + public void RelativeManagementUrl_IsRejected() + { + AssertFailsWith(Valid(o => o.Applications[0].Instances[0].ManagementUrl = "/admin"), "ManagementUrl"); + } + + [Fact] + public void OmittedManagementUrl_IsAllowed() + { + Assert.True(Run(Valid(o => o.Applications[0].Instances[0].ManagementUrl = null)).Succeeded); + } + + [Theory] + [InlineData(0)] + [InlineData(-1)] + public void NonPositiveGlobalIntervals_AreRejected(int value) + { + AssertFailsWith(Valid(o => o.PollIntervalSeconds = value), "PollIntervalSeconds"); + AssertFailsWith(Valid(o => o.TimeoutSeconds = value), "TimeoutSeconds"); + AssertFailsWith(Valid(o => o.StaleAfterSeconds = value), "StaleAfterSeconds"); + } + + [Fact] + public void StaleWindowNotGreaterThanPollInterval_IsRejected() + { + AssertFailsWith(Valid(o => o.StaleAfterSeconds = o.PollIntervalSeconds), "StaleAfterSeconds"); + AssertFailsWith(Valid(o => o.StaleAfterSeconds = o.PollIntervalSeconds - 1), "StaleAfterSeconds"); + } + + [Fact] + public void StaleWindowIsCheckedOnEffectiveValues_NotGlobals() + { + // The globals here are perfectly sane (10 / 45). Only the INSTANCE override inverts the + // relationship, which a globals-only check would wave straight through. + AssertFailsWith( + Valid(o => o.Applications[0].Instances[0].PollIntervalSeconds = 120), + "effective StaleAfterSeconds"); + } + + [Fact] + public void ApplicationLevelOverrideCanSatisfyTheStaleRule() + { + // The mirror of the previous test: an override that RESTORES the relationship must pass, so + // the rule is not just "any override fails". + var options = Valid(o => + { + o.Applications[0].Instances[0].PollIntervalSeconds = 120; + o.Applications[0].Instances[0].StaleAfterSeconds = 300; + }); + + Assert.True(Run(options).Succeeded); + } + + [Fact] + public void AllFailuresAreAccumulated_NotJustTheFirst() + { + // OptionsValidatorBase accumulates; an operator fixing a registry one boot at a time is the + // failure mode this avoids. + var result = Run(Valid(o => + { + o.Applications[0].Name = string.Empty; + o.Applications[0].Instances[0].BaseUrl = "not-a-url"; + o.PollIntervalSeconds = 0; + })); + + Assert.True(result.Failed); + Assert.True(result.Failures.Count() >= 3, $"expected >= 3 failures, got {result.Failures.Count()}"); + } + + [Theory] + [InlineData(null, null, null, 10, 3, 45)] // all defaults + [InlineData(20, null, null, 20, 3, 45)] // application override + [InlineData(20, 30, null, 30, 3, 45)] // instance beats application + [InlineData(null, 30, 90, 30, 90, 45)] // independent fields resolve independently + public void EffectiveTimings_ResolveInstanceOverApplicationOverGlobal( + int? applicationPoll, int? instancePoll, int? instanceTimeout, + int expectedPoll, int expectedTimeout, int expectedStale) + { + var options = Valid(o => + { + o.Applications[0].PollIntervalSeconds = applicationPoll; + o.Applications[0].Instances[0].PollIntervalSeconds = instancePoll; + o.Applications[0].Instances[0].TimeoutSeconds = instanceTimeout; + }); + + var timings = EffectiveTimings.Resolve(options, options.Applications[0], options.Applications[0].Instances[0]); + + Assert.Equal(TimeSpan.FromSeconds(expectedPoll), timings.PollInterval); + Assert.Equal(TimeSpan.FromSeconds(expectedTimeout), timings.Timeout); + Assert.Equal(TimeSpan.FromSeconds(expectedStale), timings.StaleAfter); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPageTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPageTests.cs new file mode 100644 index 0000000..361de76 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPageTests.cs @@ -0,0 +1,251 @@ +using Bunit; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// The page as a whole: the KPI strip an operator scans first, the group headers that carry the +/// split-brain warning, and the pause control — which has to freeze the view without pretending +/// the view is still live. +/// +public class OverviewPageTests : TestContext +{ + private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero); + + private readonly OverviewSnapshotStore _store = new(); + private readonly FakeTimeProvider _time = new(Now); + + public OverviewPageTests() + { + Services.AddSingleton(_store); + Services.AddSingleton(_time); + } + + private static InstanceSnapshot Node( + string name, + InstanceStatus? status, + string? group = null, + string? leader = null, + DateTimeOffset? lastPolled = null) => + new() + { + Application = "ScadaBridge", + Instance = name, + Group = group, + BaseUrl = $"http://{name}:8084", + HasActiveRole = group is not null, + Status = status, + LastPolledUtc = lastPolled ?? Now, + LastReachableUtc = lastPolled ?? Now, + StaleAfter = TimeSpan.FromSeconds(45), + Report = leader is null + ? null + : System.Text.Json.JsonSerializer.Deserialize( + HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader))), + }; + + private static OverviewSnapshot Snapshot(params InstanceSnapshot[] instances) + { + var applications = instances + .GroupBy(i => i.Application, StringComparer.Ordinal) + .Select(g => new ApplicationSnapshot(g.Key, g.ToList(), LeaderResolver.Resolve(g.ToList()))) + .ToList(); + + return new OverviewSnapshot(Now, applications); + } + + private IRenderedComponent RenderPage() => + RenderComponent(); + + [Fact] + public void EmptyStore_RendersAnExplicitEmptyState() + { + // Before the first sweep. It must say so rather than render a KPI strip full of zeroes, + // which would read as "the whole fleet is registered and nothing is up". + var page = RenderPage(); + + Assert.NotNull(page.Find("[data-testid=empty-registry]")); + Assert.Empty(page.FindAll(".agg-grid")); + } + + [Fact] + public void KpiStrip_CountsEachStatus() + { + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up), + Node("b", InstanceStatus.Up), + Node("c", InstanceStatus.Degraded), + Node("d", InstanceStatus.Down), + Node("e", InstanceStatus.Unreachable))); + + var page = RenderPage(); + + Assert.Equal("2", page.Find("[data-testid=kpi-up]").TextContent); + Assert.Equal("1", page.Find("[data-testid=kpi-degraded]").TextContent); + Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent); + Assert.Equal("1", page.Find("[data-testid=kpi-unreachable]").TextContent); + Assert.Equal("0", page.Find("[data-testid=kpi-stale]").TextContent); + } + + [Fact] + public void StaleInstances_CountAsStaleAndNotAlsoAsUp() + { + // Double-counting would make the KPI totals exceed the instance count and would let a + // frozen dashboard keep reporting a healthy fleet. + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up), + Node("b", InstanceStatus.Up, lastPolled: Now.AddMinutes(-5)))); + + var page = RenderPage(); + + Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent); + Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent); + } + + [Fact] + public void GroupHeader_NamesTheLeader() + { + const string leader = "akka.tcp://scadabridge@a:8082"; + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up, "site-a", leader), + Node("b", InstanceStatus.Up, "site-a", leader))); + + Assert.Contains("Leader · a", RenderPage().Markup, StringComparison.Ordinal); + } + + [Fact] + public void GroupHeader_WarnsOnDisagreement() + { + // The one output on this page that means "go look right now". + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@a:8082"), + Node("b", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@b:8082"))); + + Assert.Contains("Split brain", RenderPage().Markup, StringComparison.Ordinal); + } + + [Fact] + public void AgreeingGroup_ShowsNoWarning() + { + const string leader = "akka.tcp://scadabridge@a:8082"; + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up, "site-a", leader), + Node("b", InstanceStatus.Up, "site-a", leader))); + + Assert.DoesNotContain("Split brain", RenderPage().Markup, StringComparison.Ordinal); + } + + [Fact] + public void PublishedSnapshot_ReRendersThePage() + { + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + var page = RenderPage(); + Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent); + + _store.Publish(Snapshot(Node("a", InstanceStatus.Down))); + + page.WaitForAssertion(() => Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent)); + Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent); + } + + [Fact] + public void Paused_StopsAdoptingNewSnapshots() + { + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + var page = RenderPage(); + + page.Find("[data-testid=pause-toggle]").Click(); + _store.Publish(Snapshot(Node("a", InstanceStatus.Down))); + + Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent); + Assert.Contains("Resume auto-refresh", page.Markup, StringComparison.Ordinal); + } + + [Fact] + public void Resumed_AdoptsWhateverTheStoreHoldsNow() + { + // Resume must not replay the snapshots missed while paused — it must jump to current. + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + var page = RenderPage(); + page.Find("[data-testid=pause-toggle]").Click(); + _store.Publish(Snapshot(Node("a", InstanceStatus.Down))); + + page.Find("[data-testid=pause-toggle]").Click(); + + Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent); + } + + [Fact] + public void PausedView_AgesIntoStaleOnItsOwn() + { + // This is why pause freezes adoption rather than stopping the clock: a paused dashboard + // greys out by itself, so it cannot be mistaken for a live one. + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + var page = RenderPage(); + page.Find("[data-testid=pause-toggle]").Click(); + + _time.Advance(TimeSpan.FromMinutes(5)); + page.Render(); + + Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent); + Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent); + } + + [Fact] + public void Disposal_UnsubscribesFromTheStore() + { + // A leaked handler would call StateHasChanged on a disposed renderer every sweep, for the + // life of the process — one leak per page load. + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + var page = RenderPage(); + + page.Instance.Dispose(); + + var exception = Record.Exception(() => _store.Publish(Snapshot(Node("a", InstanceStatus.Down)))); + Assert.Null(exception); + } + + [Fact] + public void ApplicationSection_CarriesAnAnchorMatchingTheRailLink() + { + // The rail's application links are fragment anchors; a mismatch makes every one dead. + _store.Publish(Snapshot(Node("a", InstanceStatus.Up))); + + var section = RenderPage().Find("[data-testid=application]"); + + Assert.Equal( + ZB.MOM.WW.Overview.Components.Layout.MainLayout.Slug("ScadaBridge"), + section.GetAttribute("id")); + } + + [Fact] + public void OneCardPerRegisteredInstance() + { + _store.Publish(Snapshot( + Node("a", InstanceStatus.Up, "site-a"), + Node("b", InstanceStatus.Up, "site-a"), + Node("gw", InstanceStatus.Up))); + + Assert.Equal(3, RenderPage().FindAll("[data-testid=instance-card]").Count); + } + + [Fact] + public void UngroupedInstances_RenderAfterTheClusterGroups() + { + _store.Publish(Snapshot( + Node("gw", InstanceStatus.Up), + Node("a", InstanceStatus.Up, "site-a"))); + + // Enumerated, not indexed: bunit 1.40 was built against AngleSharp 1.2 and its + // RefreshableElementCollection indexer calls an IHtmlCollection member whose signature + // changed by 1.5.2 — the version this project pins for GHSA reasons. LINQ goes through + // IEnumerable and is unaffected. + var cards = RenderPage() + .FindAll("[data-testid=instance-card]") + .Select(c => c.GetAttribute("data-instance")) + .ToList(); + + Assert.Equal(["a", "gw"], cards); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs new file mode 100644 index 0000000..0da8a69 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewPollerServiceTests.cs @@ -0,0 +1,433 @@ +using System.Net; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Overview.Observability; +using ZB.MOM.WW.Overview.Polling; +using ZB.MOM.WW.Overview.Registry; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Sweep behaviour, driven directly rather than through the timer: the rules worth pinning are what +/// a sweep probes, what it does with the answers, and when it declines to probe at all. +/// +public class OverviewPollerServiceTests +{ + private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero); + + private static OverviewOptions Registry(params ApplicationEntry[] applications) => + new() + { + PollIntervalSeconds = 10, + TimeoutSeconds = 3, + StaleAfterSeconds = 45, + Applications = applications, + }; + + private static ApplicationEntry App(string name, params InstanceEntry[] instances) => + new() { Name = name, ManagementLabel = "AdminUI", Instances = instances }; + + private static InstanceEntry Node( + string name, + string baseUrl, + string? group = null, + bool hasActiveRole = false, + int? pollIntervalSeconds = null) => + new() + { + Name = name, + BaseUrl = baseUrl, + Group = group, + HasActiveRole = hasActiveRole, + PollIntervalSeconds = pollIntervalSeconds, + }; + + private static (OverviewPollerService Poller, OverviewSnapshotStore Store) Build( + OverviewOptions options, + HttpMessageHandler handler, + TimeProvider timeProvider) + { + var store = new OverviewSnapshotStore(); + var poller = new OverviewPollerService( + Options.Create(options), + new StubHttpClientFactory(handler), + store, + timeProvider, + new OverviewMetrics(new DummyMeterFactory(), store), + new PollCycleHeartbeat(timeProvider), + NullLogger.Instance); + + return (poller, store); + } + + [Fact] + public async Task Sweep_ProbesReadyForEveryInstance() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var (poller, _) = Build( + Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))), + handler, + new FakeTimeProvider(Start)); + + var snapshot = await poller.SweepAsync(); + + Assert.Equal(2, snapshot.AllInstances.Count()); + Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status)); + Assert.Equal(2, handler.Requests.Count); + } + + [Fact] + public async Task Sweep_TrailingSlashInBaseUrl_DoesNotDoubleTheSeparator() + { + // "//health/ready" would 404 on every instance — a registry typo class that must not matter. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080/"))), handler, new FakeTimeProvider(Start)); + + var snapshot = await poller.SweepAsync(); + + Assert.Equal(InstanceStatus.Up, Assert.Single(snapshot.AllInstances).Status); + } + + [Fact] + public async Task Sweep_ActiveTierProbedOnlyForInstancesThatHaveTheRole() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Status("http://a:8080/health/active", HttpStatusCode.OK) + .Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var (poller, _) = Build( + Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true), Node("gw", "http://gw:5120"))), + handler, + new FakeTimeProvider(Start)); + + var snapshot = await poller.SweepAsync(); + + Assert.DoesNotContain("http://gw:5120/health/active", handler.Requests, StringComparer.Ordinal); + var byName = snapshot.AllInstances.ToDictionary(i => i.Instance, StringComparer.Ordinal); + Assert.Equal(ActiveState.Active, byName["a"].Active); + Assert.Equal(ActiveState.NotApplicable, byName["gw"].Active); + } + + [Fact] + public async Task Sweep_StandbyAnswers503OnActive_AndIsStillUp() + { + // 503 on the ACTIVE tier is the standby's correct answer and says nothing about health — + // conflating the two tiers would paint every healthy standby in the fleet as Down. + var handler = new RoutingHandler() + .Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Status("http://b:8080/health/active", HttpStatusCode.ServiceUnavailable); + var (poller, _) = Build( + Registry(App("App", Node("b", "http://b:8080", "pair", hasActiveRole: true))), + handler, + new FakeTimeProvider(Start)); + + var instance = Assert.Single((await poller.SweepAsync()).AllInstances); + + Assert.Equal(InstanceStatus.Up, instance.Status); + Assert.Equal(ActiveState.Standby, instance.Active); + } + + [Fact] + public async Task Sweep_UnreachableInstance_SkipsTheActiveProbe() + { + // Spending a second full timeout to confirm what the first one established would double + // the worst-case sweep duration for every dead node in the registry. + var handler = new RoutingHandler().Fails("http://a:8080/health/ready", "Connection refused"); + var (poller, _) = Build( + Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))), + handler, + new FakeTimeProvider(Start)); + + await poller.SweepAsync(); + + Assert.DoesNotContain("http://a:8080/health/active", handler.Requests, StringComparer.Ordinal); + } + + [Fact] + public async Task Sweep_SickInstanceStillGetsItsActiveProbe() + { + // A 503 on ready means the app is talking. Whether the sick node is the ACTIVE half is + // precisely what decides whether this is an outage or a standby nobody is using. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy")) + .Status("http://a:8080/health/active", HttpStatusCode.OK); + var (poller, _) = Build( + Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))), + handler, + new FakeTimeProvider(Start)); + + var instance = Assert.Single((await poller.SweepAsync()).AllInstances); + + Assert.Equal(InstanceStatus.Down, instance.Status); + Assert.Equal(ActiveState.Active, instance.Active); + } + + [Fact] + public async Task Sweep_FirstFailureAfterAGoodPoll_IsDampedThenAdopted() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var time = new FakeTimeProvider(Start); + var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time); + + Assert.Equal(InstanceStatus.Up, Assert.Single((await poller.SweepAsync()).AllInstances).Status); + + handler.Fails("http://a:8080/health/ready", "Connection refused"); + + time.Advance(TimeSpan.FromSeconds(10)); + var damped = Assert.Single((await poller.SweepAsync()).AllInstances); + Assert.Equal(InstanceStatus.Up, damped.Status); + Assert.Equal(1, damped.ConsecutiveFailures); + + time.Advance(TimeSpan.FromSeconds(10)); + var adopted = Assert.Single((await poller.SweepAsync()).AllInstances); + Assert.Equal(InstanceStatus.Unreachable, adopted.Status); + Assert.Equal(2, adopted.ConsecutiveFailures); + } + + [Fact] + public async Task Sweep_LastReachableSurvivesAFailedPoll() + { + // "last seen" must keep pointing at the last real contact, not reset to null the moment + // the instance goes away — that timestamp is how an operator judges how long it has been out. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var time = new FakeTimeProvider(Start); + var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time); + + await poller.SweepAsync(); + handler.Fails("http://a:8080/health/ready", "Connection refused"); + time.Advance(TimeSpan.FromSeconds(10)); + + var instance = Assert.Single((await poller.SweepAsync()).AllInstances); + + Assert.Equal(Start, instance.LastReachableUtc); + Assert.Equal(Start.AddSeconds(10), instance.LastPolledUtc); + } + + [Fact] + public async Task Sweep_BeforeAnInstanceIsDue_LeavesItAlone() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var time = new FakeTimeProvider(Start); + var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time); + + await poller.SweepAsync(); + time.Advance(TimeSpan.FromSeconds(1)); + await poller.SweepAsync(); + + Assert.Single(handler.Requests); + } + + [Fact] + public async Task Sweep_PerInstanceInterval_IsHonoured() + { + // The override is only real if a slow instance actually skips ticks while a fast one runs. + var handler = new RoutingHandler() + .Json("http://fast:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Json("http://slow:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var time = new FakeTimeProvider(Start); + var (poller, _) = Build( + Registry(App( + "App", + Node("fast", "http://fast:8080", pollIntervalSeconds: 5), + Node("slow", "http://slow:8080", pollIntervalSeconds: 30))), + handler, + time); + + for (var i = 0; i < 4; i++) + { + await poller.SweepAsync(); + time.Advance(TimeSpan.FromSeconds(5)); + } + + Assert.Equal(4, handler.Requests.Count(r => r.Contains("fast", StringComparison.Ordinal))); + Assert.Equal(1, handler.Requests.Count(r => r.Contains("slow", StringComparison.Ordinal))); + } + + [Fact] + public async Task Sweep_DueCheckToleratesAnEarlyTick() + { + // A tick arriving a few ms early must not push the instance to half its configured rate. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var time = new FakeTimeProvider(Start); + var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time); + + await poller.SweepAsync(); + time.Advance(TimeSpan.FromMilliseconds(9_900)); + await poller.SweepAsync(); + + Assert.Equal(2, handler.Requests.Count); + } + + [Fact] + public async Task Sweep_PublishesToTheStore() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start)); + OverviewSnapshot? notified = null; + store.Changed += s => notified = s; + + var returned = await poller.SweepAsync(); + + Assert.Same(returned, store.Current); + Assert.Same(returned, notified); + } + + [Fact] + public async Task Sweep_CarriesRegistryMetadataOntoEveryCard() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Status("http://a:8080/health/active", HttpStatusCode.OK); + var application = App("ScadaBridge", Node("central-a", "http://a:8080", "central", hasActiveRole: true)); + application.Instances[0].ManagementUrl = "http://a:9000/"; + var (poller, _) = Build(Registry(application), handler, new FakeTimeProvider(Start)); + + var instance = Assert.Single((await poller.SweepAsync()).AllInstances); + + Assert.Equal("ScadaBridge", instance.Application); + Assert.Equal("central", instance.Group); + Assert.Equal("AdminUI", instance.ManagementLabel); + Assert.Equal("http://a:9000/", instance.ManagementUrl); + Assert.Equal(TimeSpan.FromSeconds(45), instance.StaleAfter); + } + + [Fact] + public async Task Sweep_ResolvesGroupLeadersIntoTheSnapshot() + { + const string leader = "akka.tcp://scadabridge@a:8082"; + var handler = new RoutingHandler() + .Json("http://a:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader))) + .Status("http://a:8084/health/active", HttpStatusCode.OK) + .Json("http://b:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader))) + .Status("http://b:8084/health/active", HttpStatusCode.ServiceUnavailable); + var (poller, _) = Build( + Registry(App( + "ScadaBridge", + Node("site-a-a", "http://a:8084", "site-a", hasActiveRole: true), + Node("site-a-b", "http://b:8084", "site-a", hasActiveRole: true))), + handler, + new FakeTimeProvider(Start)); + + var application = Assert.Single((await poller.SweepAsync()).Applications); + var group = Assert.Single(application.Groups); + + Assert.Equal("site-a", group.Name); + Assert.Equal("site-a-a", group.LeaderInstance); + Assert.False(group.Disagreement); + } + + [Fact] + public async Task Sweep_WorstStatusRollsUpAcrossTheApplication() + { + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Json("http://b:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy")); + var (poller, _) = Build( + Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))), + handler, + new FakeTimeProvider(Start)); + + var application = Assert.Single((await poller.SweepAsync()).Applications); + + Assert.Equal(InstanceStatus.Down, application.Worst); + } + + [Fact] + public async Task Sweep_OneSlowInstanceDoesNotDelayTheOthers() + { + // The fan-out must be genuinely parallel: a serial sweep over a fleet with several dead + // nodes would take timeout × dead-node-count and starve the whole dashboard. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")) + .Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var options = Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))); + var gate = new SemaphoreSlim(0); + var (poller, _) = Build(options, new GatedHandler(handler, gate), new FakeTimeProvider(Start)); + + var sweep = poller.SweepAsync(); + Assert.False(sweep.IsCompleted); + + // Both probes must already be in flight; releasing one at a time proves neither waited. + gate.Release(2); + var snapshot = await sweep.WaitAsync(TimeSpan.FromSeconds(10)); + + Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status)); + } + + [Fact] + public async Task Sweep_UnregisteredInstances_RenderAsPendingNotAsHealthy() + { + // Between host start and the first probe the registry is already published so the page has + // its layout. Those cards must read "pending", never an unmeasured "Up". + var handler = new RoutingHandler(); + var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start)); + + Assert.Empty(store.Current.Applications); + + handler.Fails("http://a:8080/health/ready", "Connection refused"); + var instance = Assert.Single((await poller.SweepAsync()).AllInstances); + + Assert.Equal(InstanceStatus.Unreachable, instance.Status); + } + + [Fact] + public void PendingSnapshot_IsNeverStale() + { + // Staleness on a card that has never been polled would be meaningless and alarming. + var pending = Snapshots.Instance("a", status: null); + + Assert.False(pending.IsStale(Start.AddYears(1))); + } + + [Fact] + public void Snapshot_GoesStaleOnceItsWindowElapses() + { + var polled = Snapshots.Instance("a") with { LastPolledUtc = Start, StaleAfter = TimeSpan.FromSeconds(45) }; + + Assert.False(polled.IsStale(Start.AddSeconds(45))); + Assert.True(polled.IsStale(Start.AddSeconds(46))); + } + + [Fact] + public async Task Sweep_HostShutdown_AbandonsTheSweepInsteadOfRecordingFailures() + { + // Writing "Unreachable" across the registry on the way out would leave a lie in the store. + var handler = new RoutingHandler() + .Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy")); + var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start)); + using var shutdown = new CancellationTokenSource(); + await shutdown.CancelAsync(); + + await Assert.ThrowsAnyAsync(() => poller.SweepAsync(shutdown.Token)); + + Assert.Empty(store.Current.Applications); + } + + [Fact] + public async Task EmptyRegistry_StartsAndStopsWithoutProbing() + { + var (poller, _) = Build(Registry(), new RoutingHandler(), new FakeTimeProvider(Start)); + + await poller.StartAsync(CancellationToken.None); + await poller.StopAsync(CancellationToken.None); + } + + /// Holds every request until the test releases it, proving the fan-out is concurrent. + private sealed class GatedHandler(RoutingHandler inner, SemaphoreSlim gate) : HttpMessageHandler + { + protected override async Task SendAsync( + HttpRequestMessage request, CancellationToken cancellationToken) + { + await gate.WaitAsync(cancellationToken).ConfigureAwait(false); + return await inner.HandleAsync(request).ConfigureAwait(false); + } + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewSnapshotStoreTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewSnapshotStoreTests.cs new file mode 100644 index 0000000..86cc357 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/OverviewSnapshotStoreTests.cs @@ -0,0 +1,97 @@ +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// The store is the cache-first seam: a page reads it and never probes. These tests pin that it +/// always has something to serve and that subscribers hear about replacements. +/// +public class OverviewSnapshotStoreTests +{ + private static OverviewSnapshot SnapshotAt(int second) => + new(new DateTimeOffset(2026, 7, 24, 12, 0, second, TimeSpan.Zero), []); + + [Fact] + public void Current_BeforeAnyPublish_IsEmptyNotNull() + { + // A page can render before the poller's first sweep finishes; a null here would be a + // NullReferenceException on the landing page of a monitoring dashboard. + var store = new OverviewSnapshotStore(); + + Assert.NotNull(store.Current); + Assert.Empty(store.Current.Applications); + } + + [Fact] + public void Publish_ReplacesCurrent() + { + var store = new OverviewSnapshotStore(); + var snapshot = SnapshotAt(1); + + store.Publish(snapshot); + + Assert.Same(snapshot, store.Current); + } + + [Fact] + public void Publish_NotifiesSubscribersWithTheNewSnapshot() + { + var store = new OverviewSnapshotStore(); + OverviewSnapshot? received = null; + store.Changed += s => received = s; + + var snapshot = SnapshotAt(2); + store.Publish(snapshot); + + Assert.Same(snapshot, received); + } + + [Fact] + public void Publish_AfterUnsubscribe_DoesNotNotify() + { + // Blazor circuits come and go; a handler that outlives its component would call + // StateHasChanged on a disposed renderer on every sweep. + var store = new OverviewSnapshotStore(); + var calls = 0; + void Handler(OverviewSnapshot _) => calls++; + + store.Changed += Handler; + store.Publish(SnapshotAt(1)); + store.Changed -= Handler; + store.Publish(SnapshotAt(2)); + + Assert.Equal(1, calls); + } + + [Fact] + public void Publish_Null_Throws() + { + Assert.Throws(() => new OverviewSnapshotStore().Publish(null!)); + } + + [Fact] + public async Task Current_ReadDuringPublish_ObservesOneWholeSnapshot() + { + // The tearing guarantee: a reader always sees a complete graph, never a half-swapped one. + var store = new OverviewSnapshotStore(); + var a = SnapshotAt(1); + var b = SnapshotAt(2); + store.Publish(a); + + using var stop = new CancellationTokenSource(); + var reader = Task.Run(() => + { + while (!stop.IsCancellationRequested) + { + var seen = store.Current; + Assert.True(ReferenceEquals(seen, a) || ReferenceEquals(seen, b)); + } + }); + + for (var i = 0; i < 1_000; i++) + store.Publish(i % 2 == 0 ? b : a); + + await stop.CancelAsync(); + await reader; + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs new file mode 100644 index 0000000..8ac9f18 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/PollingTestSupport.cs @@ -0,0 +1,197 @@ +using System.Diagnostics.Metrics; +using System.Net; +using System.Text; +using System.Text.Json; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// A meter factory with no listener attached, for tests that only need the poller to run. +internal sealed class DummyMeterFactory : IMeterFactory +{ + private readonly List _meters = []; + + public Meter Create(MeterOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + // Stamp ourselves as the scope, exactly as the real factory does. Meter names are global, + // so a MeterListener that matched on name alone would pick up meters created by tests + // running in parallel; the scope is what makes one test's instruments identifiable. + options.Scope = this; + + var meter = new Meter(options); + _meters.Add(meter); + return meter; + } + + public void Dispose() + { + foreach (var meter in _meters) + meter.Dispose(); + + _meters.Clear(); + } +} + +/// A clock the test drives by hand. +/// +/// Only is overridden: the poller's sweep — the part with rules worth +/// pinning — is exercised directly, so no test here needs a fake timer. +/// +internal sealed class FakeTimeProvider(DateTimeOffset start) : TimeProvider +{ + private DateTimeOffset _now = start; + + public override DateTimeOffset GetUtcNow() => _now; + + public void Advance(TimeSpan by) => _now += by; +} + +/// Serves canned responses per absolute URL and counts what was requested. +internal sealed class RoutingHandler : HttpMessageHandler +{ + private readonly Dictionary> _routes = new(StringComparer.OrdinalIgnoreCase); + + public List Requests { get; } = []; + + public RoutingHandler Json(string url, HttpStatusCode status, string body) + { + _routes[url] = () => new HttpResponseMessage(status) + { + Content = new StringContent(body, Encoding.UTF8, "application/json"), + }; + return this; + } + + /// + /// A status-code-only response with an EMPTY body — what the active tier actually returns when + /// it is a bare gate endpoint rather than a full health report. + /// + public RoutingHandler Status(string url, HttpStatusCode status) + { + _routes[url] = () => new HttpResponseMessage(status) + { + Content = new StringContent(string.Empty, Encoding.UTF8, "application/json"), + }; + return this; + } + + public RoutingHandler Fails(string url, string message) + { + _routes[url] = () => throw new HttpRequestException(message); + return this; + } + + /// Lets a wrapping handler delegate here without re-entering the protected member. + public Task HandleAsync(HttpRequestMessage request) => + SendAsync(request, CancellationToken.None); + + protected override Task SendAsync( + HttpRequestMessage request, CancellationToken cancellationToken) + { + var url = request.RequestUri!.ToString(); + + lock (Requests) + Requests.Add(url); + + if (!_routes.TryGetValue(url, out var factory)) + return Task.FromException(new HttpRequestException($"no route for {url}")); + + try + { + return Task.FromResult(factory()); + } + catch (HttpRequestException ex) + { + return Task.FromException(ex); + } + } +} + +/// Hands every named client the same test handler. +internal sealed class StubHttpClientFactory(HttpMessageHandler handler) : IHttpClientFactory +{ + public HttpClient CreateClient(string name) => + new(handler, disposeHandler: false) { Timeout = Timeout.InfiniteTimeSpan }; +} + +/// Builds canonical health bodies so tests state intent, not JSON. +internal static class HealthBody +{ + public static string Ready(string status, params (string Name, string Status, string? Leader)[] entries) + { + var payload = new + { + status, + totalDurationMs = 1.5, + entries = entries.ToDictionary( + e => e.Name, + e => (object)new + { + status = e.Status, + description = $"{e.Name} says {e.Status}", + durationMs = 0.5, + data = e.Leader is null ? null : new Dictionary { ["leader"] = e.Leader }, + }, + StringComparer.Ordinal), + }; + + return JsonSerializer.Serialize(payload); + } + + /// + /// Builds a body whose entries carry arbitrary data, for the cases where the key under + /// test is not leader — notably the active tier's activeNode. + /// + public static string WithData(string status, params (string Name, string Status, Dictionary? Data)[] entries) + { + var payload = new + { + status, + totalDurationMs = 1.5, + entries = entries.ToDictionary( + e => e.Name, + e => (object)new + { + status = e.Status, + description = $"{e.Name} says {e.Status}", + durationMs = 0.5, + data = e.Data, + }, + StringComparer.Ordinal), + }; + + return JsonSerializer.Serialize(payload); + } +} + +/// Builds values for the pure-logic tests. +internal static class Snapshots +{ + public static InstanceSnapshot Instance( + string name, + string? group = null, + string baseUrl = "http://host:8080", + bool hasActiveRole = true, + InstanceStatus? status = InstanceStatus.Up, + string? leader = null, + string clusterCheckName = "akka-cluster") => + new() + { + Application = "App", + Instance = name, + Group = group, + BaseUrl = baseUrl, + HasActiveRole = hasActiveRole, + Status = status, + Report = leader is null ? null : ReportWithLeader(leader, clusterCheckName), + StaleAfter = TimeSpan.FromSeconds(45), + }; + + private static ZbHealthReport ReportWithLeader(string leader, string clusterCheckName) + { + var body = HealthBody.Ready("Healthy", (clusterCheckName, "Healthy", leader)); + return JsonSerializer.Deserialize(body)!; + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/StatusDerivationTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/StatusDerivationTests.cs new file mode 100644 index 0000000..e551214 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/StatusDerivationTests.cs @@ -0,0 +1,155 @@ +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Status derivation and two-strike flap damping. These rules decide what an operator sees, so +/// every branch is pinned: the difference between Down and Unreachable sends someone to a +/// different place entirely, and damping that fires the wrong way either hides a real outage or +/// cries wolf on every dropped packet. +/// +public class StatusDerivationTests +{ + private static HealthProbeResult Reachable(string reportStatus, int statusCode) => + new(true, statusCode, new ZbHealthReport { Status = reportStatus }, TimeSpan.FromMilliseconds(5), null); + + private static HealthProbeResult Unreachable(string error) => + new(false, null, null, TimeSpan.FromMilliseconds(5), error); + + [Theory] + [InlineData("Healthy", 200, InstanceStatus.Up)] + [InlineData("Degraded", 200, InstanceStatus.Degraded)] + [InlineData("Unhealthy", 503, InstanceStatus.Down)] + public void Observe_MapsCanonicalStatuses(string reportStatus, int code, InstanceStatus expected) + { + Assert.Equal(expected, StatusDerivation.Observe(Reachable(reportStatus, code))); + } + + [Fact] + public void Observe_TransportFailure_IsUnreachable_NotDown() + { + // The distinction is the whole point: Down means "the app answered and said it is sick", + // Unreachable means "nothing answered" — usually VPN, firewall or a registry typo. + Assert.Equal(InstanceStatus.Unreachable, StatusDerivation.Observe(Unreachable("Connection refused"))); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("Weird")] + public void Observe_UnrecognisedStatus_IsUnreachable(string? reportStatus) + { + // Parsed as JSON but not this contract — something else is on that port. + Assert.Equal(InstanceStatus.Unreachable, StatusDerivation.Observe(Reachable(reportStatus!, 200))); + } + + [Theory] + [InlineData(false, 200, ActiveState.NotApplicable)] + [InlineData(false, 503, ActiveState.NotApplicable)] + public void ObserveActive_WithoutActiveRole_IsNotApplicable(bool hasRole, int code, ActiveState expected) + { + Assert.Equal(expected, StatusDerivation.ObserveActive(hasRole, Reachable("Healthy", code))); + } + + [Theory] + [InlineData(200, ActiveState.Active)] + [InlineData(503, ActiveState.Standby)] + [InlineData(404, ActiveState.Unknown)] + public void ObserveActive_MapsStatusCodes(int code, ActiveState expected) + { + Assert.Equal(expected, StatusDerivation.ObserveActive(true, Reachable("Healthy", code))); + } + + [Fact] + public void ObserveActive_NoAnswer_IsUnknown_NotStandby() + { + // Guessing Standby on a failed probe would let a pair render with zero Active nodes during + // a blip, or — worse, if guessed the other way — with two. + Assert.Equal(ActiveState.Unknown, StatusDerivation.ObserveActive(true, Unreachable("timed out"))); + Assert.Equal(ActiveState.Unknown, StatusDerivation.ObserveActive(true, probe: null)); + } + + [Fact] + public void Damp_FirstFailureFromGoodState_HoldsThePreviousStatus() + { + var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 0, InstanceStatus.Unreachable); + + Assert.Equal(InstanceStatus.Up, status); + Assert.Equal(1, failures); + } + + [Fact] + public void Damp_SecondConsecutiveFailure_Flips() + { + var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 1, InstanceStatus.Unreachable); + + Assert.Equal(InstanceStatus.Unreachable, status); + Assert.Equal(2, failures); + } + + [Fact] + public void Damp_RecoveryIsImmediate_EvenAfterManyFailures() + { + var (status, failures) = StatusDerivation.Damp(InstanceStatus.Down, 27, InstanceStatus.Up); + + Assert.Equal(InstanceStatus.Up, status); + Assert.Equal(0, failures); + } + + [Fact] + public void Damp_FirstEverPollThatFails_IsAdoptedImmediately() + { + // No previous state means nothing to protect and nothing better to show. Holding "Up" here + // would be inventing a healthy reading that was never observed. + var (status, failures) = StatusDerivation.Damp(previous: null, 0, InstanceStatus.Unreachable); + + Assert.Equal(InstanceStatus.Unreachable, status); + Assert.Equal(1, failures); + } + + [Fact] + public void Damp_AlternatingFailures_NeverFlip() + { + // The flapping link this exists for: fail, recover, fail, recover. The counter must reset + // on every good poll, so the card stays Up instead of oscillating. + InstanceStatus? status = InstanceStatus.Up; + var failures = 0; + + foreach (var observed in new[] + { + InstanceStatus.Unreachable, InstanceStatus.Up, + InstanceStatus.Unreachable, InstanceStatus.Up, + InstanceStatus.Unreachable, InstanceStatus.Up, + }) + { + (var next, failures) = StatusDerivation.Damp(status, failures, observed); + status = next; + } + + Assert.Equal(InstanceStatus.Up, status); + Assert.Equal(0, failures); + } + + [Fact] + public void Damp_DegradedIsNotAFailure_AndIsAdoptedImmediately() + { + // Degraded is a real answer from a live app, not a failed probe: it must show at once, and + // must not accumulate strikes toward Down. + var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 1, InstanceStatus.Degraded); + + Assert.Equal(InstanceStatus.Degraded, status); + Assert.Equal(0, failures); + } + + [Fact] + public void Damp_TwoDifferentFailureKindsInARow_StillFlips() + { + // Down then Unreachable is two consecutive failures, not one of each kind restarting the + // count — the instance is not answering healthily either way. + var (first, failures) = StatusDerivation.Damp(InstanceStatus.Up, 0, InstanceStatus.Down); + Assert.Equal(InstanceStatus.Up, first); + + var (second, _) = StatusDerivation.Damp(first, failures, InstanceStatus.Unreachable); + Assert.Equal(InstanceStatus.Unreachable, second); + } +} diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZB.MOM.WW.Overview.Tests.csproj b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZB.MOM.WW.Overview.Tests.csproj new file mode 100644 index 0000000..dbd6fb4 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZB.MOM.WW.Overview.Tests.csproj @@ -0,0 +1,33 @@ + + + + + false + + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZbHealthReportClientTests.cs b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZbHealthReportClientTests.cs new file mode 100644 index 0000000..7e33ae8 --- /dev/null +++ b/ZB.MOM.WW.Overview/tests/ZB.MOM.WW.Overview.Tests/ZbHealthReportClientTests.cs @@ -0,0 +1,252 @@ +using System.Net; +using System.Text; +using ZB.MOM.WW.Overview.Polling; + +namespace ZB.MOM.WW.Overview.Tests; + +/// +/// Golden-payload tests for the health client. The fixtures are hand-written to the exact shape +/// ZbHealthWriter emits — including an entry WITH data (0.2.0+) and one WITHOUT +/// (every check that publishes none, and every pre-0.2.0 node), because tolerating both is what +/// lets the dashboard run against a fleet that is only partly bumped. +/// +public class ZbHealthReportClientTests +{ + private const string ReadyHealthyWithData = """ + { + "status": "Healthy", + "totalDurationMs": 12.34, + "entries": { + "akka-cluster": { + "status": "Healthy", + "description": "Akka cluster member status: Up", + "durationMs": 1.23, + "data": { + "selfAddress": "akka.tcp://scadabridge@site-a-a:8082", + "selfRoles": ["Site", "site-a"], + "memberCount": 2, + "unreachableCount": 0, + "leader": "akka.tcp://scadabridge@site-a-a:8082" + } + }, + "localdb": { + "status": "Healthy", + "description": "Site LocalDb reachable.", + "durationMs": 0.45 + } + } + } + """; + + private const string ReadyUnhealthyNoData = """ + { + "status": "Unhealthy", + "totalDurationMs": 3001.5, + "entries": { + "database": { + "status": "Unhealthy", + "description": null, + "durationMs": 3000.1 + } + } + } + """; + + private static ZbHealthReportClient ClientReturning( + HttpStatusCode statusCode, + string body, + string contentType = "application/json") => + new(new HttpClient(new StubHandler((statusCode, body, contentType)))); + + private static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(3); + + [Fact] + public async Task Probe_HealthyBody_ParsesEnvelopeAndEntries() + { + var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + + Assert.True(result.Reachable); + Assert.Equal(200, result.StatusCode); + Assert.Equal("Healthy", result.Report!.Status); + Assert.Equal(12.34, result.Report.TotalDurationMs); + Assert.Equal(new[] { "akka-cluster", "localdb" }, result.Report.Entries.Keys.OrderBy(k => k, StringComparer.Ordinal)); + Assert.Equal("Site LocalDb reachable.", result.Report.Entries["localdb"].Description); + } + + [Fact] + public async Task Probe_EntryWithData_ExposesLeaderAndCounts() + { + var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + var entry = result.Report!.Entries["akka-cluster"]; + + Assert.Equal("akka.tcp://scadabridge@site-a-a:8082", entry.DataString("leader")); + Assert.Equal(2, entry.DataInt("memberCount")); + Assert.Equal(0, entry.DataInt("unreachableCount")); + } + + [Fact] + public async Task Probe_EntryWithoutData_ReadsAsNullNotAsAnError() + { + // The 0.1.0-era / no-data case. It must degrade to "no leader shown", never to a parse + // failure — that tolerance is what keeps the dashboard usable mid-rollout. + var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + var entry = result.Report!.Entries["localdb"]; + + Assert.True(result.Reachable); + Assert.Null(entry.Data); + Assert.Null(entry.DataString("leader")); + Assert.Null(entry.DataInt("memberCount")); + } + + [Fact] + public async Task Probe_503_IsAParseableAnswer_NotAFailedProbe() + { + // 503 carries a full body naming the failing check — the single most useful payload the + // dashboard receives. Treating it as a transport failure would throw that away. + var client = ClientReturning(HttpStatusCode.ServiceUnavailable, ReadyUnhealthyNoData); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + + Assert.True(result.Reachable); + Assert.Equal(503, result.StatusCode); + Assert.Equal("Unhealthy", result.Report!.Status); + Assert.Null(result.Report.Entries["database"].Description); + } + + [Theory] + [InlineData("502 Bad Gateway")] + [InlineData("not json at all")] + public async Task Probe_NonJsonBody_IsUnreachable(string body) + { + var client = ClientReturning(HttpStatusCode.OK, body, contentType: "text/html"); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + + Assert.False(result.Reachable); + Assert.Null(result.Report); + Assert.NotNull(result.Error); + } + + [Fact] + public async Task Probe_ConnectionRefused_IsUnreachableAndDoesNotThrow() + { + var client = new ZbHealthReportClient( + new HttpClient(new ThrowingHandler(new HttpRequestException("Connection refused")))); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + + Assert.False(result.Reachable); + Assert.Contains("refused", result.Error, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task Probe_Timeout_IsUnreachableAndReportsTheTimeout() + { + var client = new ZbHealthReportClient(new HttpClient(new HangingHandler())); + + var result = await client.ProbeAsync("http://node/health/ready", TimeSpan.FromMilliseconds(50)); + + Assert.False(result.Reachable); + Assert.Contains("timed out", result.Error, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task Probe_HostShutdown_PropagatesCancellation() + { + // A shutdown must NOT be recorded as a timed-out instance — the sweep is being abandoned, + // and writing "Unreachable" for every node on the way out would be a lie in the snapshot. + using var shutdown = new CancellationTokenSource(); + await shutdown.CancelAsync(); + var client = new ZbHealthReportClient(new HttpClient(new HangingHandler())); + + await Assert.ThrowsAnyAsync( + () => client.ProbeAsync("http://node/health/ready", ProbeTimeout, shutdown.Token)); + } + + [Theory] + [InlineData(HttpStatusCode.OK, 200)] + [InlineData(HttpStatusCode.ServiceUnavailable, 503)] + public async Task ProbeStatus_EmptyBody_IsStillAnAnswer(HttpStatusCode status, int expected) + { + // The active tier answers with a status code; the body is not part of that contract. An + // empty 200/503 is the endpoint telling us the role plainly, and insisting on parseable + // JSON here reports every such node as "role unknown". + var client = ClientReturning(status, string.Empty); + + var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout); + + Assert.True(result.Reachable); + Assert.Equal(expected, result.StatusCode); + Assert.Null(result.Report); + Assert.Null(result.Error); + } + + [Fact] + public async Task ProbeStatus_StillParsesABodyWhenOneIsThere() + { + var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData); + + var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout); + + Assert.Equal("Healthy", result.Report!.Status); + } + + [Fact] + public async Task ProbeStatus_TransportFailure_IsStillUnreachable() + { + // Relaxing the body requirement must not relax what "reachable" means. + var client = new ZbHealthReportClient( + new HttpClient(new ThrowingHandler(new HttpRequestException("Connection refused")))); + + var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout); + + Assert.False(result.Reachable); + Assert.Null(result.StatusCode); + } + + [Fact] + public async Task Probe_ReadyTierStillDemandsAParseableBody() + { + // The ready tier's body IS the payload — the per-check list and the leader data come from + // it. An empty 200 there means something other than a family app is on that port. + var client = ClientReturning(HttpStatusCode.OK, string.Empty); + + var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout); + + Assert.False(result.Reachable); + } + + private sealed class StubHandler((HttpStatusCode Status, string Body, string ContentType) response) + : HttpMessageHandler + { + protected override Task SendAsync( + HttpRequestMessage request, CancellationToken cancellationToken) => + Task.FromResult(new HttpResponseMessage(response.Status) + { + Content = new StringContent(response.Body, Encoding.UTF8, response.ContentType), + }); + } + + private sealed class ThrowingHandler(Exception exception) : HttpMessageHandler + { + protected override Task SendAsync( + HttpRequestMessage request, CancellationToken cancellationToken) => + Task.FromException(exception); + } + + private sealed class HangingHandler : HttpMessageHandler + { + protected override async Task SendAsync( + HttpRequestMessage request, CancellationToken cancellationToken) + { + await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken).ConfigureAwait(false); + return new HttpResponseMessage(HttpStatusCode.OK); + } + } +} diff --git a/akka_failover.md b/akka_failover.md new file mode 100644 index 0000000..fb6c847 --- /dev/null +++ b/akka_failover.md @@ -0,0 +1,531 @@ +# Akka.NET HA, Failover & Singletons — ScadaBridge + OtOpcUa Deep Dive + +*Written 2026-07-22 against ScadaBridge `main @ 69b3ccfc` and OtOpcUa (`lmxopcua`) `master @ 0de1352e`. +Both trees contain the 2026-07-21 downing/election corrections (ScadaBridge `cf3bd52f`; OtOpcUa `2964361a` auto-down + `50b55ee4` oldest-driver election — note `0de1352e` itself is the adjacent NU1903 dependency pin, not the redundancy fix).* + +**Design priority (stated 2026-07-22):** for the 2-node site pairs, a network-partition +split-brain is *less* risky than failing to bring the second node up quickly. HA goal: +either node can be launched first, automatic failover, no manual intervention. The +architecture below is evaluated against that priority throughout. + +--- + +## 1. How Akka.NET handles failover — the fundamentals both apps build on + +The deployment unit at every site is a **2-node Akka.NET cluster** (one node per Windows +VM). Central is also a 2-node cluster, with SQL Server; sites have no SQL Server. +Everything HA-related decomposes into five Akka mechanisms: + +### 1.1 Membership and seed nodes +A cluster forms by gossip. Each node lists `seed-nodes`; a booting node sends `InitJoin` +to the seeds. **Only the FIRST seed in the list may join itself to form a *new* cluster** — +any other node waits until some existing member answers. Both apps set +`min-nr-of-members = 1`, so a single node (if it's allowed to form) goes `Up` alone and +hosts all singletons — the pair does not need both members to become operational. + +### 1.2 Failure detection (the heartbeat) +Every node heartbeats its peers and feeds the intervals into a **Phi Accrual failure +detector**. When the suspicion level crosses the threshold, the peer is marked +**Unreachable** — a suspicion, not a removal. Both apps run the same values: + +| Setting | Value (both apps) | Meaning | +|---|---|---| +| `failure-detector.heartbeat-interval` | **2 s** | Cluster-membership heartbeat rate | +| `failure-detector.threshold` | 10.0 | Phi threshold (suspicion sensitivity) | +| `failure-detector.acceptable-heartbeat-pause` | **10 s** | Tolerated silence before Unreachable | +| `transport-failure-detector` (remoting layer) | 2 s / 10 s pause | Detects dead TCP associations | + +So a hard-killed peer is declared Unreachable roughly **10–12 s** after death. These are +exactly the "heartbeat with a small adjustable timeout" — in ScadaBridge they are bound +from config (`Cluster:HeartbeatInterval`, `Cluster:FailureDetectionThreshold` → +`ClusterOptions`, rendered into HOCON by `AkkaHostedService.BuildHocon`); in OtOpcUa they +live in `akka.conf` (lines 68–72) in the Cluster library. + +### 1.3 Downing (the decision to act) +Unreachable alone changes nothing — the leader cannot make membership decisions while any +member is unreachable, singletons do not move, the cluster is wedged. Something must +**Down** the unreachable member. This is the strategy choice, and it is the whole +availability-vs-partition-safety trade: + +- **Auto-downing** (`Akka.Cluster.AutoDowning` + `auto-down-unreachable-after`): after the + window, the leader *among the reachable members* downs the unreachable one. In a 1-vs-1 + split, **each side downs the other** — the crash case fails over perfectly; a genuine + partition produces dual-active. +- **Split Brain Resolver `keep-oldest` (+ `down-if-alone`)**: keeps the side containing + the oldest member. Partition-safe, but **fatally broken for a 2-node pair**: in + Akka.NET 1.5.62, `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side + holding ≥ 2 members (`otherSide == 1 && thisSide >= 2`). A 1-vs-1 survivor whose peer + (the oldest) crashed takes `DownReachable` and **downs itself** — proven live: + `SBR took decision Akka.Cluster.SBR.DownReachable … including myself, [1] unreachable of [2] members`. + Net effect: the pair could not survive a crash of the oldest node — total outage. + +**Both apps switched their default from `keep-oldest` to `auto-down` on 2026-07-21** +(ScadaBridge `cf3bd52f`, decision record +`ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`; OtOpcUa +`2964361a`, same fix ported). The owner rationale recorded in the decision doc is +verbatim the priority stated above: *"network partitions are less of a risk than if this +stops working."* + +Rejected alternatives (from the decision record): `static-quorum(1)` → `DownAll` (worse); +`static-quorum(2)` → survivor self-downs; `keep-majority` → keeps the lowest-address side, +same fatal crash re-keyed; `lease-majority` → needs a shared lease store (no SQL/K8s at +sites); a third arbiter node → no third VM at sites; a custom downing provider → would +reimplement AutoDowning. + +### 1.4 Cluster singletons (what actually "fails over") +`ClusterSingletonManager` runs on every node of the (optionally role-filtered) cluster +and guarantees exactly one instance of the actor, hosted on the **oldest member** of that +role. `ClusterSingletonProxy` routes to wherever it currently lives. Failover semantics: + +- **Graceful shutdown** (service stop, redeploy): hand-over protocol — the old host + transfers to the new oldest with no dead window beyond hand-over latency. Both apps + hook `CoordinatedShutdown` (`run-by-clr-shutdown-hook = on`; ScadaBridge additionally + drains each singleton via a `PhaseClusterLeave` `GracefulStop` task, 10 s per singleton, + `phases.cluster-leave.timeout = 15s`). +- **Hard crash**: the singleton is simply gone until the dead node is **Downed**; then the + new oldest member's manager instantiates a fresh instance. This is why downing is the + linchpin — no downing decision, no singleton recovery, ever. + +### 1.5 Oldest-member semantics — the family-wide rule +Both codebases converged on the same insight: **Akka's "leader" (lowest address) and +"oldest member" (singleton placement) diverge permanently after any node restart** — the +restarted node rejoins as a *new incarnation*, becoming the youngest, but may still have +the lowest address and thus be leader. Any "active node" decision keyed on +leader/lowest-address can therefore point at a node that is *not* hosting the singletons. +Both apps now derive "active/primary" strictly from **oldest Up member** (ScadaBridge +`ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa `RedundancyStateActor.SelectDriverPrimary` +with `Member.AgeOrdering`), which by construction matches singleton placement. + +--- + +## 2. The failover timeline (and the knobs that tune it) + +End-to-end failover after a hard node death, with current production values: + +``` +t=0 node dies (process crash / VM loss) +t≈10-12s failure detector: peer marked Unreachable ← acceptable-heartbeat-pause (10s) +t≈25-27s auto-down window expires; survivor downs peer ← auto-down-unreachable-after (15s) +t≈25-30s survivor is now oldest; singletons instantiate; + primary/active gates flip; ServiceLevel 250 moves +``` + +Measured, not theoretical: ScadaBridge's 2026-07-21 docker drill measured **28 s** +(active-node crash → standby takeover) and **27 s** (standby crash → removal, zero routing +blips); the in-process `FailoverTimingTests` measured 33.7 s for a full cycle. + +**Tuning for faster failover** (per the availability-first priority): + +| Knob | Where | Current | Effect of lowering | +|---|---|---|---| +| `acceptable-heartbeat-pause` | SB: `Cluster:FailureDetectionThreshold`; OtOpcUa: `akka.conf` | 10 s | Faster Unreachable, but < ~5 s risks false positives from GC pauses / Windows VM scheduling stalls / snapshot freezes | +| `heartbeat-interval` | SB: `Cluster:HeartbeatInterval`; OtOpcUa: `akka.conf` | 2 s | More samples → detector converges faster; cheap to lower to 1 s | +| Auto-down window | SB: `Cluster:StableAfter`; OtOpcUa: `DowningStableAfter` const (15 s) + emitted HOCON | 15 s | Direct 1:1 reduction of failover time. Every second cut is a second of dual-active risk during a *transient* blip (link flap, VM pause) — the window exists to let transients heal before the irreversible Down | +| `down-removal-margin` (OtOpcUa akka.conf) | 15 s | Delay before rebalancing after removal | Keep ≈ downing window | + +Constraint pinned by an OtOpcUa test (`Downing_window_exceeds_the_acceptable_heartbeat_pause`): +the downing window **must exceed** `acceptable-heartbeat-pause`, otherwise a node can be +downed on a pause the detector was configured to tolerate. A realistic aggressive profile +for the site pairs would be ~5 s pause + ~7–8 s window ≈ **12–13 s total failover**, +at the cost of downing a peer that stalls (not dies) for > ~12 s — acceptable per the +stated priority, but worth validating against real Windows VM GC/backup/snapshot pauses +before rollout. In ScadaBridge these are plain per-deployment config values; in OtOpcUa +the 15 s window is currently a constant in `ServiceCollectionExtensions.cs` (only the +*strategy* is config-selectable), so making the window configurable would be a small +change there. + +**What happens to the dead node's side:** both apps set +`run-coordinated-shutdown-when-down = on` — a node that gets Downed (e.g. it was merely +partitioned and the other side downed it) terminates its own ActorSystem. Both apps run a +termination watchdog (ScadaBridge in `AkkaHostedService`; OtOpcUa +`ActorSystemTerminationWatchdog`) that distinguishes intentional stop from self-down and +calls `StopApplication()`, so the **process exits and the service supervisor restarts it** +(Windows: `sc.exe failure … restart/5000/restart/30000/restart/60000`; Docker: +`restart: unless-stopped`). The restarted process rejoins as a fresh incarnation — +youngest member, i.e. it comes back as the **standby**, never usurping the survivor. + +--- + +## 3. ScadaBridge + +### 3.1 Cluster formation +All HOCON is generated in code — `AkkaHostedService.BuildHocon(...)` +(`src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:258–329`) from +`ClusterOptions`/`NodeOptions` (`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`). +Key facts: + +- **Seed nodes**: central docker pair → `scadabridge-central-a:8081` then + `…-central-b:8081` (both nodes list `central-a` first). Site pairs analogous on 8082. +- **`MinNrOfMembers = 1`** (validator *requires* exactly 1) — one node forms the cluster + and brings up all singletons. +- **Roles**: central node = `["Central"]`; site node = `["Site", "site-{SiteId}"]` + (`BuildRoles`, line 406). The role drives the entire composition branch in `Program.cs`. +- **Ports**: central remoting 8081; site remoting 8082, gRPC 8083, metrics 8084. + +### 3.2 Downing configuration +Configurable via **`Cluster:SplitBrainResolverStrategy`**, allowed values `auto-down` | +`keep-oldest` (case-insensitive; validator `ClusterOptionsValidator`). **Default and all +16 appsettings files: `auto-down`.** The HOCON branch (`AkkaHostedService.cs:275–286`): + +```csharp +var downingBlock = string.Equals(strategy, "auto-down", OrdinalIgnoreCase) + ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster"" + auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}" + : /* SBR provider + split-brain-resolver { active-strategy, stable-after, + keep-oldest { down-if-alone } } */; +``` + +`StableAfter = 00:00:15` doubles as the SBR `stable-after` and the auto-down window. +`DownIfAlone` is now validated only under `keep-oldest` (inert under auto-down). +`keep-oldest` remains supported and SBR-path tests still pin it +(`SbrFailoverTests.HardCrashOfYoungerNode_SbrDownsIt_AndOldestKeepsSingleton`). + +### 3.3 Cluster singletons +All registered through `SingletonRegistrar.Start(...)` (names `{name}-singleton` / +`{name}-proxy`, PoisonPill termination, optional role, per-singleton +`PhaseClusterLeave` drain task of 10 s). + +**Central — 7 singletons** (role not further scoped; central nodes are all `Central`): + +| Singleton | Actor | Purpose | +|---|---|---| +| `notification-outbox` | `NotificationOutboxActor` | Notification ingest/dispatch/purge | +| `audit-log-ingest` | `AuditLogIngestActor` | Central audit ingest from sites | +| `site-call-audit` | `SiteCallAuditActor` | SiteCalls upserts + central→site retry/discard relay | +| `audit-log-purge` | `AuditLogPurgeActor` | Daily partition-switch purge | +| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Per-site audit reconciliation pull | +| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI sampling (not readiness-gated) | +| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Expired staging-row reclaim (not readiness-gated) | + +`RequiredSingletonsHealthCheck` probes the first five via bounded `Identify` for +`/health/ready`; the last two are deliberately best-effort. + +**Site — 2 singletons**, role-scoped to `site-{SiteId}` so each site pair elects its own: + +| Singleton | Actor | Purpose | +|---|---|---| +| `deployment-manager` | `DeploymentManagerActor` | Deployment lifecycle; drains in-flight SQLite writes on hand-over | +| `event-log-handler` | `EventLogHandlerActor` | Event-log queries always served by the active node (event log is node-local, not replicated) | + +Deliberately **not** singletons: `dcl-manager` (DataConnectionManagerActor) and +`CertStoreActor` run on **every** site node — the deployment singleton fans PKI trust +changes to both nodes' cert stores so a failover never finds a stale trust store. + +### 3.4 Active-node gating +One definition of "active": `ActiveNodeEvaluator.SelfIsOldestUp` +(`src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs`) — oldest +Up member, explicitly **not** `cluster.State.Leader` (the XML docs spell out the +leader-vs-oldest divergence after restarts). Consumers: + +- **Central inbound API**: `IActiveNodeGate` → `InboundApiEndpointFilter` returns + **HTTP 503 `{code:"STANDBY_NODE"}`** on the standby; Traefik routes on + `/health/active` (`OldestNodeActiveHealthCheck` + `DatabaseHealthCheck`), so the proxy + and the API agree on which node is active. +- **Site store-and-forward**: `storeAndForwardService.SetDeliveryGate(() => + clusterNodeProvider.SelfIsPrimary)` — only the active node runs the delivery sweep; + without the gate both nodes would deliver the same replicated `sf_messages` rows + (systematic duplicate external calls). Re-evaluated per sweep tick, so delivery resumes + within one `RetryTimerInterval` after failover. +- **Heartbeat to central**: `SiteCommunicationActor` stamps `HeartbeatMessage.IsActive` + from the same evaluator (exception-safe, defaults to `false`). + +Takeover is purely membership-driven: peer Downed → survivor becomes oldest → +`SelfIsOldestUp` flips → singletons, S&F gate, `/health/active`, heartbeat all follow. + +### 3.5 Site state without SQL Server — LocalDb +Sites persist everything in **one consolidated SQLite file** (`LocalDb:Path`, required, +on the data volume) via `ZB.MOM.WW.LocalDb` — 10 replicated tables: `OperationTracking`, +`site_events`, `sf_messages` (the store-and-forward buffer), and 7 config tables +(deployed configurations, static overrides, shared scripts, external systems, DB +connections, DCL definitions, native alarm state). + +**Pair replication** (default OFF; opt-in `LocalDb:Replication:PeerAddress` + matching +`ApiKey` both sides; fail-closed auth interceptor; rides the existing gRPC 8083 listener): +CDC triggers capture row mutations, shipped bidirectionally under HLC last-writer-wins; +snapshot resync merges per row and never deletes. **This is what makes site failover +real**: the standby already holds the S&F buffer and the full config, so on takeover it +just opens its delivery gate over data it already has — zero cross-node fetch (the old +notify-and-fetch deploy path is deleted). + +Not replicated (by design): the event log (hence the `event-log-handler` singleton), +in-memory alarm state (re-evaluated from incoming values on the survivor), and +`notification_lists`/`smtp_configurations` (would ship plaintext SMTP passwords). + +**Operational constraints:** +- **Stop/start a site pair TOGETHER.** Rolling upgrades are unsupported: Phase 2 deleted + the legacy replicator and its `SfBufferSnapshot` compat handler, so a mixed-version + pair runs but silently does not converge. +- A node offline longer than `TombstoneRetention` (7 d) can **resurrect deleted rows** on + rejoin; within the window rejoin is safe and self-correcting. +- `MaxBatchSize` counts **rows, not bytes** — rig pins 16 (the 500 default could build a + ~35 MB batch against the 4 MB gRPC cap). + +### 3.6 Verified behavior + remaining gaps +- Auto-down failover drill (docker `failover-drill.sh`, SIGKILL): active-crash takeover + 28 s, standby-crash 27 s, 0 routing blips; both drill modes now *assert* recovery. +- After a real partition the two mutually-downed halves **do not merge** (quarantined + association) — an operator restarts one side, which rejoins as standby. This is the + accepted dual-active trade. +- **The old "restart central-a and central-b together (SBR self-down)" gotcha is closed** + by auto-down — but a related constraint survives with a different mechanism: the + **seed-node bootstrap constraint** (§6.1). +- ⚠ Stale doc: `docs/deployment/topology-guide.md:101` still says keep-oldest + + down-if-alone. Authoritative: `Component-ClusterInfrastructure.md`, the auto-down + decision record, `docker/README.md`. + +--- + +## 4. OtOpcUa + +### 4.1 Cluster formation +Base HOCON in the library (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf`): +system `otopcua`, remoting port 4053, `min-nr-of-members = 1`, empty seeds/roles overlaid +at runtime from `AkkaClusterOptions` (`Cluster:*` config; roles restricted to +`admin` | `driver` | `dev` by `RoleParser`). Bootstrap via Akka.Hosting: +`WithOtOpcUaClusterBootstrap` (remoting + clustering + the downing HOCON, Prepended). +Note the dual role plumbing: `OTOPCUA_ROLES` drives DI composition (`hasAdmin`/`hasDriver` +branches in `Program.cs`), while `Cluster__Roles__*` drives the actual Akka member roles — +docker-dev sets both (a past production incident came from setting only `OTOPCUA_ROLES`, +leaving the Akka member role-less). + +**Current mesh shape (important):** today OtOpcUa runs **one Akka mesh for the whole +fleet** — docker-dev is six nodes (central-1/2 = `admin,driver`; site-a-1/2, site-b-1/2 = +`driver`) all seeded by `central-1`, with logical MAIN/SITE-A/SITE-B separation done by +`ServerCluster.ClusterId` rows in the shared ConfigDb, not by separate meshes. The +per-cluster mesh design (§4.4) will change this to one mesh per 2-node pair — the shape +that matches your site deployment. + +### 4.2 Downing configuration +Same key as ScadaBridge: **`Cluster:SplitBrainResolverStrategy`**, default **`auto-down`**, +allowed `auto-down` | `keep-oldest`, anything else **fails the host at startup** +(`ArgumentOutOfRangeException` — no silent fallback; pinned by +`Unknown_strategy_throws_rather_than_silently_falling_back`). `BuildDowningHocon` emits: + +``` +akka.cluster { + downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster" + auto-down-unreachable-after = 15000ms +} +``` + +(the comment notes `auto-down-unreachable-after` defaults to `off` upstream and is +load-bearing — AutoDowning without it never downs anything). Under `keep-oldest` the +typed `KeepOldestOption { DownIfAlone = true }` installs the SBR instead; the +`split-brain-resolver` block in akka.conf is inert under the default. Tests read the +**effective config off a running ActorSystem** (not the input HOCON) — provider class, +15 s window, case-insensitivity, and the window-exceeds-heartbeat-pause invariant. + +**Live gate still open:** the 1-vs-1 oldest-crash pathology cannot be exercised on the +current six-node docker-dev mesh; the crash-the-oldest drill is deferred to the +per-cluster mesh work (Phase 6/7). Supporting live evidence exists from a 2-container +kill test (`archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`) and +from ScadaBridge's identical fix. Cautionary note recorded in `docs/Redundancy.md`: +the pre-existing `HardKillFailoverTests` stayed green under broken keep-oldest because it +shut down the *transport* (ActorSystem alive) rather than killing the process — "a +standing example of a green test over a fatal defect." + +### 4.3 Redundancy primary election + ServiceLevel +- **`RedundancyStateActor`** (an `admin`-role singleton) computes the fleet snapshot: + `SelectDriverPrimary` = oldest **Up** member carrying the `driver` role + (`Member.AgeOrdering`; `Leaving` excluded — a leaving node is handing its singletons + over and must not be Primary). Debounced 250 ms, republished on a 10 s heartbeat over + distributed pub/sub topic `redundancy-state` (DPS doesn't replay to late subscribers). +- **`NodeRedundancyState`** carries `RedundancyRole { Primary, Secondary, Detached }` and + `IsDriverPrimary` (renamed from `IsRoleLeaderForDriver` — "the name now describes what + the value means rather than how it used to be computed"). +- **ServiceLevel** (`ServiceLevelCalculator`): health basis 240 (healthy) / 200 (stale) / + 100 (critically degraded) / 0 (down), **+10 if driver Primary**, so a healthy pair + reads **250 / 240**. Published into the OPC UA `Server.ServiceLevel` variable via + `SdkServiceLevelPublisher` (deferred publisher until the SDK server exists; first + computed value always published so no node lingers at the SDK default 255). +- **Client-visible failover is OPC UA non-transparent redundancy**: each node has its own + `ApplicationUri`; clients read `Server.ServerArray` (self + peers, from + `PeerApplicationUris`) and pick the endpoint with the highest ServiceLevel. The shipped + client (`OpcUaClientService`) does this: failover URL list, KeepAlive-failure trigger, + single-flight failover guard, data + alarm subscription replay after reconnect. + **Consumer note:** because Primary = oldest (not lowest address), after any node + restart a ServiceLevel-selecting client may legitimately prefer a different node than + before — that's correct behavior, not a bug. + +**Primary gating of the data plane (`PrimaryGatePolicy`)** — the S4 default-deny fix: + +```csharp +localRole switch { + Primary => true, + Secondary or Detached => false, + _ /* unknown */ => driverMemberCount <= 1, // multi-driver: DENY until proven +}; +``` + +During the boot window (role not yet known) a multi-driver node **rejects device writes** +with `not primary (role unknown)` (denial meter `reason=role-unknown`); a single-driver +node stays default-allow. Gated surfaces: device writes, native-alarm acks, alarm-alert +fan-out. A deliberately *different* policy governs alarm-history drain: unknown → drain +anyway ("a duplicate row beats a silent gap"), keyed on whether the Primary shares this +node's queue (pair-local). Live-proven on a 2-node docker rig (R2-04 T15): ServiceLevel +250/240, boot-window write rejected, steady-state secondary rejected + reverted, primary +write reaches the device. + +Failover sequence when the primary dies: peer downed (auto-down) → survivor is oldest +driver → `SelectDriverPrimary` flips → snapshot republished → gates open on the survivor → +ServiceLevel 250 moves → OPC UA clients see the old endpoint die (KeepAlive) and re-select +by ServiceLevel. Singletons migrate to the same node the election names, because both +derive from member age. + +### 4.4 Singletons and the per-cluster mesh gap +Five control-plane singletons, all role-pinned to `admin`: + +| Singleton | Actor | Purpose | +|---|---|---| +| `config-publish` | `ConfigPublishCoordinator` | Persists per-node deploy ACKs | +| `admin-operations` | `AdminOperationsActor` | Operator ack/shelve; TagConfig deploy gate | +| `audit-writer` | `AuditWriterActor` | Audit persistence | +| `fleet-status` | `FleetStatusBroadcaster` | AdminUI fleet panel | +| `redundancy-state` | `RedundancyStateActor` | Primary election + DPS publisher | + +**KNOWN LIMITATION:** the Primary election is scoped **per Akka cluster, not per +application Cluster**. On the one-mesh fleet, `SelectDriverPrimary` names exactly ONE +Primary among *all* driver nodes (MAIN + SITE-A + SITE-B), while every gated resource is +pair-local. The per-cluster mesh design +(`OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`) fixes this by construction: +one Akka mesh per application Cluster (two nodes max), central↔cluster joined by explicit +transports (modeled on ScadaBridge's ClusterClient/gRPC/HTTP trio), so "am I the driver +Primary?" and "the resource I'm gating" have the same scope. Status: Phases 0a (auto-down) +and 0b (oldest-Up election) DONE and merged; Phases 1–7 (ClusterNode columns, comm actors, +config fetch-and-cache, cut driver-side ConfigDb, gRPC stream contract, mesh partition, +failover drill/live gate) not started — **but now sequenced by a program plan +(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, 2026-07-22), driven by the +decision that OtOpcUa pairs will run on the SAME Windows VMs as the ScadaBridge nodes** — +two independent, identically-postured 2-node clusters per site, and driver nodes off the +ConfigDb entirely (sites have no SQL Server). + +### 4.5 Surviving central-SQL loss — availability guarantees +Two mechanisms make a driver pair independent of central SQL Server availability: + +- **Address-space availability guarantee (#485/#486):** an artifact the node cannot read + is *no answer*, never *an empty configuration*. Previously a transient ConfigDb error + mid-deploy parsed zero bytes into an empty composition → the planner diffed it as a + `PureRemove` of every node → served address space emptied, drivers stopped, + subscriptions cleared, deploy still reported Applied. Now `OpcUaPublishActor` / + `DriverHostActor` hold last-known-good address space, drivers, and subscriptions, and + report the deploy **Failed** without advancing the revision so the retry lands. +- **LocalDb Phase 1 boot-from-cache:** every driver node caches the chunked, SHA-256 + verified deployed-configuration artifact in a pair-local SQLite DB; a node restarting + into a central-SQL outage boots from the last artifact it applied. With opt-in pair + replication (`deployment_artifacts` + `deployment_pointer`, HLC last-writer-wins, + fail-closed auth) either node holds a byte-identical copy even if it never applied that + deploy itself. Scope: config artifact only — no live tags/alarms/historian data. + +--- + +## 5. Dual-active: what it concretely means here + +Under auto-down, a *genuine* partition (both nodes alive, link cut) ends with each side +downing the other and running active until an operator restarts one side (the mutual +downing quarantines the association — the sides will not self-merge). Accepted per the +availability-first decision; concrete blast radius: + +**ScadaBridge site pair:** both nodes' `SelfIsOldestUp` = true → both open the S&F +delivery gate → **duplicate external deliveries** of the same replicated `sf_messages` +rows for the partition's duration; both stamp `IsActive` heartbeats. LocalDb replication +itself is partition-tolerant (HLC last-writer-wins reconverges on heal, after the +operator restart). Central pair: both serve `/health/active` = 200 and run duplicate +singleton sets against the same SQL Server — duplicated sweeps/purges (mostly idempotent +upserts), and Traefik may alternate between two "active" backends. + +**OtOpcUa pair:** both nodes elect themselves driver Primary → both publish +**ServiceLevel 250** → clients on either side keep their session; both sides accept +device writes → **two masters against the same field devices** for the duration. This is +the classic OT dual-primary risk — mitigated by rarity (2 VMs, one LAN segment, real +partitions ≈ switch failure) and by the alternative (keep-oldest) having been proven to +turn a plain crash into a total outage. + +Detection/ops: the survivor logs the downing decision; OtOpcUa emits +`otopcua.redundancy.service_level_change` and `primary_gate_denied` meters; two +simultaneous 250s / two 200-responses on `/health/active` are the alarm signal. Recovery +is always "restart one side; it rejoins as youngest/standby." + +--- + +## 6. Gaps vs the stated HA goal ("either node first, automatic failover") + +### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo) +**Automatic failover: achieved.** Either node crashing is survivable, unattended, in +~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin` +forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but +execution overturned the original design and the two repos legitimately diverged: + +**⚠️ The original design's safety claim was FALSE — proven independently in both repos.** +The planned `SelfFormAfter` watchdog assumed "window expires mid-join-handshake → benign, +Akka ignores `Join` once joined." Wrong: the watchdog sits *outside* Akka's join +handshake and cannot distinguish *no seed answered* from *a seed answered and the join is +in flight*. On a routine restart into a live peer, the join stalls briefly behind removal +of the node's own stale incarnation; a `Join(self)` fired in that window **abandons the +in-flight join and forms a second cluster at the same address — a permanent island**. +ScadaBridge proved it with a written test (split still unhealed after 90 s); OtOpcUa hit +it live when its manual-failover drill bounced a node (InitJoinAck received, no Welcome +inside the window, fallback fired → second cluster). + +**ScadaBridge's fix — self-first seed ordering (no runtime code).** Every node lists +ITSELF as `seed-nodes[0]`, partner second. Akka's own `FirstSeedNodeProcess` then +implements exactly the intended semantics *inside* the handshake (InitJoin the other +seeds; self-join only if nobody answers) — race-free by construction. A `StartupValidator` +rule hard-gates the ordering at boot. Live-proven behaviors: lone cold-start self-joins +after ~5 s (`seed-node-timeout`); restart into a live peer rejoins (never islands); +simultaneous reachable cold-start converges to ONE cluster (the old "two clusters that +never merge" objection was disproved by test); a genuine boot partition dual-forms — the +accepted auto-down trade. There is **no `SelfFormAfter` option in ScadaBridge** — the +whole watchdog was rejected in code review. + +**OtOpcUa's fix — converged on self-first ordering the same day.** OtOpcUa first shipped +the watchdog and hit the race live (its manual-failover drill bounced a node; the fallback +islanded it), patched it with a TCP reachability guard — and then **retired the whole +watchdog hours later** (`lmxopcua master @ 3f24d4d6`, a breaking change): +`ClusterBootstrapFallback` and `Cluster:SelfFormAfter` are **deleted**, docker-dev +`central-2` now lists itself as `SeedNodes__0`, and a new `AkkaClusterOptionsValidator` +enforces the ordering at boot. Two subtleties its rule handles that ScadaBridge's didn't +need: the invariant is **conditional** (self must be `seed[0]` only IF self appears in +the list at all — site nodes seed off central only and are deliberately not seeds), and +identity compares on **`PublicHostname` + port** (the address Akka puts in `SelfAddress`), +because matching the `0.0.0.0` bind address would leave the rule silently inert on the +whole rig. The retirement was proven by a falsifiability control: the old peer-first +ordering test failed at 11 s against the watchdog — demonstrating the watchdog could +never usefully fire for the nodes that needed it. + +**Result: both repos now share the identical mechanism** — self-first seed ordering, +validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence +item is already done; what remains there is only the per-pair seed topology itself. + +**Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/` +overlay must have its `SeedNodes` reordered self-first before its next deploy — the new +validator rule is a deliberate hard boot gate. + +### 6.2 Other open items +- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to + per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven. +- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land + (§4.4) — only relevant while multiple pairs share one mesh. +- **OtOpcUa's 15 s downing window is a code constant**, not config — worth making + tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is: + `Cluster:StableAfter`). +- **ScadaBridge site pairs: stop/start together** for upgrades (no mixed-version LocalDb + replication); ≥ 7-day-offline nodes risk tombstone resurrection. +- **Stale doc**: `ScadaBridge/docs/deployment/topology-guide.md:101` still describes + keep-oldest. + +--- + +## 7. Quick reference — file map + +| Concern | ScadaBridge | OtOpcUa | +|---|---|---| +| HOCON / downing emit | `Host/Actors/AkkaHostedService.cs:258–329` | `Cluster/ServiceCollectionExtensions.cs:119–146` + `Cluster/Resources/akka.conf` | +| Options + validation | `ClusterInfrastructure/ClusterOptions.cs`, `ClusterOptionsValidator.cs` | `Cluster/AkkaClusterOptions.cs`, `RoleParser.cs` | +| Strategy key | `Cluster:SplitBrainResolverStrategy` (default `auto-down`) | same key, same default; unknown value = startup failure | +| Active/primary definition | `Communication/ClusterState/ActiveNodeEvaluator.cs` (oldest Up) | `ControlPlane/Redundancy/RedundancyStateActor.cs` (oldest Up `driver`) | +| Data-plane gating | `InboundApiEndpointFilter` (503 STANDBY_NODE), S&F `SetDeliveryGate` | `Runtime/Drivers/PrimaryGatePolicy.cs` + `DriverHostActor` gates | +| Singleton registration | `Host/Actors/SingletonRegistrar.cs` (7 central + 2 site) | `ControlPlane/ServiceCollectionExtensions.cs:44–94` (5, role `admin`) | +| Client-visible failover | Traefik on `/health/active`; gRPC A/B flip | OPC UA ServiceLevel 250/240 + `Server.ServerArray`; client KeepAlive failover | +| Self-down recovery | watchdog in `AkkaHostedService` → process exit → supervisor | `Host/ActorSystemTerminationWatchdog.cs` → same | +| Decision records | `docs/plans/2026-07-21-auto-down-availability-decision.md`, `docs/requirements/Component-ClusterInfrastructure.md` | `docs/Redundancy.md` §"Split-brain / downing", `docs/plans/2026-07-21-per-cluster-mesh-design.md` | +| Failover drills / tests | `docker/failover-drill.sh` (28 s/27 s), `SbrFailoverTests`, `FailoverTimingTests` | `SplitBrainResolverActivationTests`, `RedundancyPrimaryElectionTests`, R2-04 T13/T15 live gate | diff --git a/components/health/GAPS.md b/components/health/GAPS.md index df5590a..33c0d86 100644 --- a/components/health/GAPS.md +++ b/components/health/GAPS.md @@ -17,6 +17,28 @@ reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 part > until someone re-walks them. They also predate HistorianGateway, which is absent from every > table below. +> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0` +> (2026-07-24).** Everything below about the active-node check describes the leader-based design and +> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule. +> +> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed +> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no +> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a +> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently +> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the +> lower address it becomes leader while the singletons stay on the other node. During a partition +> both sides also compute themselves leader. +> +> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to +> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`). +> The result was that the package's whole active-node surface had **zero consumers** — avoided, not +> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check +> and the gate on it, and both apps migrated back and deleted their copies. +> +> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the +> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its +> lowest-addressed one. The library now pins exactly that fixture. + ## Divergence vs spec ### §1 Endpoint tiers diff --git a/components/health/shared-contract/ZB.MOM.WW.Health.md b/components/health/shared-contract/ZB.MOM.WW.Health.md index 7686d95..3ee6cd6 100644 --- a/components/health/shared-contract/ZB.MOM.WW.Health.md +++ b/components/health/shared-contract/ZB.MOM.WW.Health.md @@ -10,7 +10,7 @@ exist: OtOpcUa `Health/` (three-tier + probes), ScadaBridge `Health/` (inline pr ``` ZB.MOM.WW.Health # core: tier convention, response writer, IActiveNodeGate, GrpcDependencyHealthCheck -ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ActiveNodeHealthCheck, AkkaActiveNodeGate +ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ClusterActiveNode, ActiveNodeHealthCheck, AkkaActiveNodeGate ZB.MOM.WW.Health.EntityFrameworkCore # DatabaseHealthCheck ``` @@ -79,7 +79,7 @@ public static class ZbHealthWriter public static Task WriteJsonAsync(HttpContext context, HealthReport report); } -/// Single-property seam: is this node the active/leader node? +/// Single-property seam: is this node the active node (oldest Up member)? /// Attach to route groups via RequireActiveNode(). Implement with AkkaActiveNodeGate (Health.Akka) /// or a project-specific implementation for non-Akka nodes. public interface IActiveNodeGate @@ -162,49 +162,77 @@ public sealed class AkkaClusterStatusPolicy public static AkkaClusterStatusPolicy OtOpcUaCompat { get; } } -/// Checks whether this node is the designated leader / active node. -/// Optional role parameter scopes the check to nodes carrying that role. -/// Register to tag ZbHealthTags.Active. +/// THE family-wide definition of "the active node": the OLDEST Up member, optionally scoped to a +/// role — the member ClusterSingletonManager places singletons on. Never cluster leadership, which is +/// address-ordered and diverges from singleton placement permanently after any restart (0.3.0). +public static class ClusterActiveNode +{ + /// The oldest Up member, optionally scoped to a role. Null when none is Up. + public static Member? OldestUpMember(IEnumerable members, string? role = null); + public static Member? OldestUpMember(Cluster cluster, string? role = null); + + /// True when the local member is Up, carries the role (if given), and no eligible member is + /// older. Identity compares UniqueAddress, so a node restarted on the same host:port is a + /// different member during the overlap. + public static bool SelfIsActive(Cluster cluster, string? role = null); + + /// The first role in rolePreference that selfRoles carries, else null. Lets a fused node answer + /// for the role its singletons are pinned to (e.g. ["admin", "driver"]). + public static string? ResolveActiveRole(IEnumerable selfRoles, IReadOnlyList rolePreference); +} + +/// Configures which role's active node a node competes to be. +public sealed class ActiveNodeHealthCheckOptions +{ + /// Candidate roles in descending precedence. Empty (default) = unscoped: oldest Up member of the + /// whole cluster. + public IReadOnlyList RolePreference { get; init; } + + /// Status when RolePreference is non-empty and the node carries none of those roles — it owns no + /// active work. Defaults to Unhealthy: the active tier exists so a load balancer can pick exactly + /// one node, and "not applicable => Healthy" made the tier answer 200 on every node (lmxopcua#494). + public HealthStatus NoActiveRoleStatus { get; init; } +} + +/// Active-tier probe: Healthy on the one node in charge, Unhealthy (503) on its standby. +/// Register to tag ZbHealthTags.Active. Selects via ClusterActiveNode (oldest Up member). +/// Result data carries selfAddress, activeRole (when scoped) and activeNode, so a standby reports +/// WHO is active. /// -/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem is not yet -/// available (e.g. during startup before Akka is initialised), the check returns Degraded rather -/// than throwing. This makes the check startup-safe. +/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem or cluster is not +/// yet available (e.g. during startup before Akka is initialised), the check returns Degraded rather +/// than throwing. This makes the check startup-safe. A reachable cluster with no Up member in scope +/// is Unhealthy, not Degraded — a cluster that never forms must not sit at 200. /// public sealed class ActiveNodeHealthCheck : IHealthCheck { - /// Role-less constructor: Healthy = node is Up AND cluster leader (ScadaBridge ActiveNode pattern). - /// Returns Degraded when ActorSystem/cluster is not yet ready. - /// - /// The application service provider. ActorSystem is resolved lazily so the check is - /// startup-safe: if no ActorSystem is registered yet the result is Degraded. - /// + /// Unscoped: the active node is the oldest Up member of the whole cluster. public ActiveNodeHealthCheck(IServiceProvider serviceProvider); - /// Role-filtered constructor: Healthy = (node lacks the role) OR (node carries role AND is role-singleton leader). - /// Degraded = node carries role but is not the role-singleton leader (OtOpcUa AdminRoleLeader pattern). - /// Returns Degraded when ActorSystem/cluster is not yet ready. - /// - /// The application service provider. ActorSystem is resolved lazily so the check is - /// startup-safe: if no ActorSystem is registered yet the result is Degraded. - /// + /// Role-scoped: the active node is the oldest Up member carrying `role`. public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role); + /// Configured, including a multi-role preference order. + public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options); + public Task CheckHealthAsync( HealthCheckContext context, CancellationToken cancellationToken = default); } -/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem -/// (SelfMember Up + cluster leader), null-guarded for startup safety. +/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem via +/// ClusterActiveNode (oldest Up member, optional role scope), null-guarded for startup safety. +/// Shares its rule with ActiveNodeHealthCheck, so a gated endpoint and /health/active agree. /// Register as a singleton. Does NOT resolve ActiveNodeHealthCheck from DI. public sealed class AkkaActiveNodeGate : IActiveNodeGate { /// /// The application service provider. ActorSystem is resolved lazily; if not yet available /// IsActiveNode returns false (safe default during startup). - /// The gate checks SelfMember.Status == Up AND cluster.State.Leader == self.Address directly. + /// The gate delegates to ClusterActiveNode.SelfIsActive(cluster, role) directly. /// - public AkkaActiveNodeGate(IServiceProvider serviceProvider); + /// Optional role scope; null (default) considers every Up member. + public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null); public bool IsActiveNode { get; } } diff --git a/components/health/spec/SPEC.md b/components/health/spec/SPEC.md index 6612904..9cc6b98 100644 --- a/components/health/spec/SPEC.md +++ b/components/health/spec/SPEC.md @@ -10,8 +10,10 @@ three code-verified current-state docs (`../current-state/`). Goal is *path to s `/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable `AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in -OtOpcUa and ScadaBridge; a role-filtered `ActiveNodeHealthCheck` that unifies OtOpcUa's -`AdminRoleLeaderHealthCheck` and ScadaBridge's `ActiveNodeHealthCheck`; a generic +OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the +`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's +`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original +leader-based check proved wrong on a live rig; a generic `DatabaseHealthCheck` that covers both apps' EF Core probe patterns; a `GrpcDependencyHealthCheck` for downstream gRPC reachability. @@ -29,7 +31,7 @@ Three tiers, always served in this order, each filtered to a named tag: | Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ | |---|---|---|---|---|---|---| | Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 | -| Active | `/health/active` | `active` | Is this the leader / active node? Fails (503) on a standby or role-member-but-not-leader node. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 | +| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 | | Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 | Notes: @@ -77,21 +79,45 @@ rather than Unhealthy. Registered to the `ready` tag. > `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result, > not to an explicit policy match. -### 2.3 Active / leader probe — `ActiveNodeHealthCheck` +### 2.3 Active-node probe — `ActiveNodeHealthCheck` -Checks whether this node is the designated leader (active node). Accepts an optional Akka -cluster role name that scopes the check to nodes carrying that role. +Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a +role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the +`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate +and the probe an orchestrator routes by cannot disagree. -**Two behaviors unify the existing divergence:** +**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is +*address*-ordered — host, then port, with no relationship to time — while singleton placement is +*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart: +the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it +becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based +active tier therefore routes traffic to a node that is not hosting the work. During a partition both +sides also compute themselves leader, so both answer 200. -| Mode | Role param | Origin | Healthy | Degraded | Unhealthy | -|---|---|---|---|---|---| -| Role-less | `null` | ScadaBridge `ActiveNodeHealthCheck` | Node is `Up` **and** cluster leader | — | Otherwise | -| Role-filtered | e.g. `"admin"` | OtOpcUa `AdminRoleLeaderHealthCheck` | Node does **not** carry the role (not a participant — ignore it) **or** node carries the role and is the role-singleton leader | Carries the role but is **not** the role-singleton leader (role member, not leader) | — | +| Mode | `RolePreference` | Healthy | Unhealthy | +|---|---|---|---| +| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise | +| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) | -The role-filtered variant maps "not a member of the role" to Healthy (transparent — the probe -is irrelevant for this node). This is the correct behavior for heterogeneous clusters where not -every node carries every role. Registered to the `active` tag. +`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to +(`admin` before `driver`), while a single-role node answers for its own role. + +**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is +deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier +exists so a load balancer can pick exactly one node, and a node owning no active work must not be in +that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200 +on **every** node and silently broke leader-pinning ([lmxopcua#494]). + +Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting +node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster +is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never +forms must not sit at 200 forever. + +Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data` +object, so a standby reports **who** is active and a dashboard can render a pair from either node's +payload alone. + +[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494 ### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck` @@ -151,9 +177,11 @@ plug it into `MapHealthChecks` options and also call it from custom endpoints. `IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses whether the current node should accept write / active-role requests. The default implementation, `AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the -`ActorSystem` is available, `SelfMember.Status == Up`, and the node is the cluster leader. It is -null-guarded and returns `false` when the `ActorSystem` is not yet ready (safe default during -startup). It does **not** resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on +`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the +same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an +orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem` +or cluster is not yet ready (safe default during startup, matching the standby case). It does **not** +resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on `IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable` on standby nodes. @@ -180,6 +208,13 @@ predicates and response writer. ## 6. Migration notes +> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance. +> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared +> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with +> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit +> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check +> on oldest-`Up`-member semantics (§2.3) — which is what they now use. + | Project | Current state | Gap | What normalizes | |---|---|---|---| | **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. | diff --git a/docs/plans/2026-07-22-initjoin-selfform-fallback.md b/docs/plans/2026-07-22-initjoin-selfform-fallback.md new file mode 100644 index 0000000..6bd6dd8 --- /dev/null +++ b/docs/plans/2026-07-22-initjoin-selfform-fallback.md @@ -0,0 +1,104 @@ +# InitJoin Self-Form Fallback (`Cluster:SelfFormAfter`) + Manual Failover Control — Shared Design & Index + +> **For Claude:** This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section): +> +> - **ScadaBridge:** `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10) +> - **OtOpcUa:** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (7 tasks + "Reserved for additions" section) +> - **OtOpcUa mesh-alignment PROGRAM (added 2026-07-22):** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`. +> +> Execute each with superpowers-extended-cc:executing-plans **in that repo**. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file. + +**Goal:** (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps. + +> ## ⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned +> +> Both repo plans executed. **The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores `Join` once joined" is FALSE** — the watchdog sits outside Akka's join handshake, and a `Join(self)` fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a **permanent second cluster at the same address**. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill. +> +> - **ScadaBridge** rejected the watchdog entirely and shipped **self-first seed ordering** (each node lists itself as `seed-nodes[0]`; Akka's `FirstSeedNodeProcess` provides the intended semantics race-free inside the handshake; `StartupValidator` hard-gates the ordering). No `SelfFormAfter` option exists there. See its plan's "Part 1 as executed". +> - **OtOpcUa** first kept `SelfFormAfter` behind a TCP reachability guard, then **converged the same day** (`lmxopcua @ 3f24d4d6`): `ClusterBootstrapFallback` + `SelfFormAfter` deleted, self-first ordering shipped with a conditional `AkkaClusterOptionsValidator` (self must be `seed[0]` only IF self is a seed — site nodes exempt; identity on `PublicHostname`+port, not the 0.0.0.0 bind). +> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI). +> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account. + +**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site. + +--- + +## Design + +### The defect being fixed + +Akka.NET only allows the **first-listed seed** to self-join and form a *new* cluster. Every other node loops on `InitJoin` forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's `ClusterOptions.SeedNodes` doc comment even claims "either can start first and form the cluster" — which the deployed configs do **not** deliver (both nodes list node-a first). The plans make the claim true and fix the comment. + +### Behavior specification + +| Scenario | Behavior with fallback | +|---|---| +| Peer alive (any boot order) | Normal seed join within ms — fallback never fires (membership arrives before the window) | +| Lone cold-start, node IS in its own seed list | After `SelfFormAfter` (default 10 s): warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) | +| Lone cold-start, node NOT in its own seed list | Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. **This is the OtOpcUa site-node topology** (seeded only by `central-1`). | +| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. | +| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. | +| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior | +| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster | + +### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`) + +- **ScadaBridge:** `ScadaBridge:Cluster:SelfFormAfter` → `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`), validated non-negative-or-null by `ClusterOptionsValidator`. +- **OtOpcUa:** `Cluster:SelfFormAfter` → `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`); disable-on-`<=0` semantics live in the fallback itself. +- NOT a code constant (deliberate — see the `DowningStableAfter` lesson, `akka_failover.md` §6.2). + +### Mechanism (both apps, deliberately duplicated like the termination watchdogs) + +`ClusterBootstrapFallback.Arm(system, options, logger)`, called at ActorSystem creation (ScadaBridge: inside `AkkaHostedService.GetOrCreateActorSystem`; OtOpcUa: an Akka.Hosting `AddStartup` task inside `WithOtOpcUaClusterBootstrap`, so production and test hosts arm identically): + +1. `SelfFormAfter` null/`≤0` → log info, return (disabled). +2. Self address not in own `SeedNodes` (parsed via `Address.Parse`, exact equality) → log info, return (island guard). +3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`. +4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`. + +Races are benign — Akka ignores `Join` once joined this incarnation. + +### Manual failover control (Part D design) + +**Mechanism — graceful `Cluster.Leave`, never `Down`.** "Trigger failover" = issue `Cluster.Leave()` from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its `CoordinatedShutdown` runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the **youngest** member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency. + +Shared rules: +- **Admin-only:** button renders inside the app's admin-policy `AuthorizeView` (ScadaBridge `AuthorizationPolicies.RequireAdmin` on `/monitoring/health`; OtOpcUa the mutating-pages policy from `AdminUiPolicies` on `/clusters/{id}/redundancy`). +- **Peer guard:** disabled (with tooltip) when fewer than 2 Up members carry the relevant role. +- **Confirmation dialog** stating exactly what happens. +- **Audited** through each app's existing audit seam before the Leave (`cluster.manual-failover`; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha). +- **UX note (ScadaBridge):** Traefik routes the UI to the *active* node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this. +- **Scope note (OtOpcUa):** until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md`). +- **Election parity:** each service's target query is intentionally identical to the app's active/primary election (`ActiveNodeEvaluator.SelfIsOldestUp` / `RedundancyStateActor.SelectDriverPrimary`), pinned by a parity test — the node acted on must be exactly the one hosting the singletons. +- **Self-form interplay:** none — the restarted node rejoins via its peer (alive by definition here), so `SelfFormAfter` never fires on this path. + +### Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html) + +**Verdict: usable in principle, deliberately NOT adopted.** The MultiNode TestKit (`Akka.MultiNode.TestAdapter` + `Akka.Remote.TestKit` + `Akka.Cluster.TestKit`) runs each `RoleName` as a separate process under `dotnet test`, with `TestConductor` able to `Blackhole` links and `Exit` nodes, synchronized by `EnterBarrier`. Reasons not to use it here: + +1. **Everything this feature needs is testable in-process** with real ActorSystems built from production bootstrap code (ScadaBridge `TwoNodeClusterFixture`; OtOpcUa `SplitBrainResolverActivationTests` pattern). Same fidelity, no new runner. +2. **Heavy project-level cost:** dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing. +3. **OtOpcUa xunit constraint:** the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3. +4. **The one MNTR-only scenario** (true boot-time partition → dual self-form) validates an already-documented *accepted* risk; the family's preferred proof is the docker rig drill (`failover-drill.sh` precedent, or `docker network disconnect`). + +**Follow-up:** revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted. + +### Out of scope (recorded follow-ups) + +- Making OtOpcUa's 15 s `DowningStableAfter` configurable — now listed in the OtOpcUa plan's "Reserved for additions". +- Failure-detector tuning knobs for faster failover (both apps) — see `akka_failover.md` §2 tuning table; OtOpcUa side listed in its reserved-additions section. +- Staggered Windows service start delays (pure ops; document only). +- `deploy/wonder-app-vd03/` gitignored production overlay (ops applies `SelfFormAfter` on next deploy). +- ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first). + +--- + +## Task C1: Family docs flip (execute AFTER both repo plans complete and merge) + +**Classification:** trivial +**Estimated implement time:** ~2 min + +**Files:** +- Modify: `~/Desktop/scadaproj/akka_failover.md` §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 `DowningStableAfter` line unless the OtOpcUa reserved-addition shipped. +- Modify: `~/Desktop/scadaproj/CLAUDE.md` — brief addition to the ScadaBridge and OtOpcUa index rows. +- Update memory `ha-availability-over-partition-safety.md` — mark the decision implemented. diff --git a/docs/plans/2026-07-22-initjoin-selfform-fallback.md.tasks.json b/docs/plans/2026-07-22-initjoin-selfform-fallback.md.tasks.json new file mode 100644 index 0000000..05681af --- /dev/null +++ b/docs/plans/2026-07-22-initjoin-selfform-fallback.md.tasks.json @@ -0,0 +1,8 @@ +{ + "planPath": "docs/plans/2026-07-22-initjoin-selfform-fallback.md", + "note": "SPLIT 2026-07-22: executable tasks moved to the per-repo plans — ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (10 tasks) and OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks + reserved additions). Only C1 executes from this file, after BOTH repo plans complete and merge.", + "tasks": [ + {"id": 0, "subject": "Task C1: Flip akka_failover.md §6.1 status + CLAUDE.md index + memory (after both repo plans merge)", "status": "pending"} + ], + "lastUpdated": "2026-07-22T00:00:00Z" +} diff --git a/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json new file mode 100644 index 0000000..33253e5 --- /dev/null +++ b/docs/plans/2026-07-22-overview-dashboard-impl-plan.md.tasks.json @@ -0,0 +1,241 @@ +{ + "plan": "docs/plans/2026-07-22-overview-dashboard-impl-plan.md", + "tasks": [ + { + "id": "1", + "subject": "0.1 Health writer: optional per-entry `data` object", + "status": "completed", + "blockedBy": [] + }, + { + "id": "2", + "subject": "0.2 AkkaClusterHealthCheck: populate cluster-view data", + "status": "completed", + "blockedBy": [] + }, + { + "id": "3", + "subject": "0.3 Health 0.2.0: version, test, pack, publish, verify", + "status": "completed", + "blockedBy": [ + "1", + "2" + ] + }, + { + "id": "4", + "subject": "1.1 ScadaBridge: bump Health packages to 0.2.0", + "status": "completed", + "blockedBy": [ + "3" + ] + }, + { + "id": "5", + "subject": "1.2 ScadaBridge: site health checks (akka-cluster, localdb, active-node)", + "status": "completed", + "blockedBy": [ + "4" + ] + }, + { + "id": "6", + "subject": "1.3 ScadaBridge: map MapZbHealth on the site pipeline", + "status": "completed", + "blockedBy": [ + "5" + ] + }, + { + "id": "7", + "subject": "1.4 ScadaBridge: site health tests", + "status": "completed", + "blockedBy": [ + "6" + ] + }, + { + "id": "8", + "subject": "2.1 OtOpcUa: verify `akka` check is the shared AkkaClusterHealthCheck", + "status": "completed", + "blockedBy": [] + }, + { + "id": "9", + "subject": "2.2 OtOpcUa: bump Health to 0.2.0", + "status": "completed", + "blockedBy": [ + "3", + "8" + ] + }, + { + "id": "10", + "subject": "2.3 Alignment bumps: mxgw + HistorianGateway Health 0.2.0", + "status": "completed", + "blockedBy": [ + "3" + ] + }, + { + "id": "11", + "subject": "3.1 Overview: scaffold project", + "status": "completed", + "blockedBy": [ + "3" + ] + }, + { + "id": "12", + "subject": "3.2 Overview: registry options + validation", + "status": "completed", + "blockedBy": [ + "11" + ] + }, + { + "id": "13", + "subject": "3.3 Overview: health client + status model", + "status": "completed", + "blockedBy": [ + "11" + ] + }, + { + "id": "14", + "subject": "3.4 Overview: poller + snapshot store", + "status": "completed", + "blockedBy": [ + "12", + "13" + ] + }, + { + "id": "15", + "subject": "3.5 Overview: leader aggregation", + "status": "completed", + "blockedBy": [ + "13" + ] + }, + { + "id": "16", + "subject": "3.6 Overview: UI (ThemeShell, Overview page, InstanceCard)", + "status": "completed", + "blockedBy": [ + "14", + "15" + ] + }, + { + "id": "17", + "subject": "3.7 Overview: self-observability", + "status": "completed", + "blockedBy": [ + "14" + ] + }, + { + "id": "18", + "subject": "3.8 Overview: tests", + "status": "completed", + "blockedBy": [ + "16", + "17" + ] + }, + { + "id": "19", + "subject": "3.9 Overview: config + docker", + "status": "completed", + "blockedBy": [ + "17" + ] + }, + { + "id": "20", + "subject": "4 Live acceptance (design \u00a79)", + "status": "pending", + "blockedBy": [ + "7", + "9", + "18", + "19" + ] + } + ], + "notes": { + "deferred": [ + "Task 2.2 rig check (admin node /health/ready shows data.leader) \u2014 live docker-dev, folded into Phase 4.", + "Phase 1 DoD rig curls (site :8084 ready/active split) \u2014 live docker rig, folded into Phase 4." + ], + "branches": { + "scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard, stacked", + "ScadaBridge": "feat/site-node-health (commit 47850c0f)", + "OtOpcUa": "feat/health-0.2.0-bump (commit 2e515c34, made in an isolated git worktree \u2014 feat/mesh-phase6 working tree untouched)", + "HistorianGateway": "chore/health-0.2.0-bump (commit 51f0c2b)", + "MxAccessGateway": "chore/health-0.2.0-bump (commit 2d54ace)" + }, + "front_loaded": [ + "Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot).", + "Poller / snapshot-store / leader-resolver / bUnit InstanceCard + Overview page tests were written with tasks 3.4-3.6; task 3.8 now covers only the WebApplicationFactory boot tests." + ], + "deviations": [ + "Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.", + "Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it.", + { + "task": "3.7", + "what": "Metric names follow components/observability/spec/METRIC-CONVENTIONS.md (dot-separated overview.instance.status / overview.poll.duration, unit s) instead of the plan's literal zb_overview_instance_status.", + "why": "That spec is marked Standardized and authoritative for the family; the Prometheus exporter renders them as overview_instance_status and overview_poll_duration_seconds. Gauge labels are application/node/group \u2014 'node', not 'instance', because Prometheus stamps its own instance label on every scraped series and a colliding metric label is silently renamed exported_instance." + }, + { + "task": "3.9", + "what": "Three registries, not one: appsettings.json ships an EMPTY Applications list, appsettings.Development.json a host-reachable subset (4 instances), appsettings.Docker.json the full 16-instance fleet.", + "why": "The dev rigs publish only some node ports to the host. OtOpcUa publishes none per-node (only Traefik :9200, load-balanced across the central pair, so it cannot identify a node) and ScadaBridge's site nodes keep :8084 container-internal. A host-run dashboard can genuinely only see part of the fleet; the containerised one joins the rig networks and sees all of it. Empty Applications in appsettings.json keeps a registry-less production deploy failing fast at ConfigPreflight instead of booting into an empty dashboard." + }, + { + "task": "3.9", + "what": "appsettings.json declares real Serilog sinks and MinimumLevel overrides rather than the family's empty \"Serilog\": {}.", + "why": "AddZbSerilog drives sinks entirely from configuration and adds none of its own, so an empty section produces an app that logs NOWHERE \u2014 verified against the running HistorianGateway container, whose log stream is completely empty for exactly this reason (OtOpcUa, which populates the section, logs normally). The HttpClient override is needed too: 16 instances on a 10s cadence emit several Information lines a second of pure probe noise." + }, + { + "task": "3.9", + "what": "Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS.", + "why": "On .NET 8+ the aspnet base image declares its binding via ASPNETCORE_HTTP_PORTS=8080, not ASPNETCORE_URLS. Clearing only URLS leaves Kestrel logging 'Overriding address(es) http://*:8080' on every boot \u2014 the URLs-override trap arriving by default rather than by mistake." + }, + { + "task": "3.9", + "what": "No container healthcheck.", + "why": "aspnet:10.0 ships neither curl nor wget, and adding a --healthcheck mode to the app would put a second entry point into production code purely to satisfy Compose. The plan explicitly allows skipping." + } + ], + "phase4_findings": [ + "Rig ports verified by probing, not by reading compose (they disagree): OtOpcUa central-1/2 serve health on :9000 but site-* on :8080 (their explicit Kestrel listeners for LocalDb sync re-bind the primary HTTP port); ScadaBridge central on :5000, sites on :8084.", + "ScadaBridge site nodes currently answer :8084/health/ready with 404 \u2014 MapZbHealth on sites is on the feat/site-node-health branch and the rig runs main. Phase 4 needs the rigs redeployed from the 0.2.0 branches.", + "No leader chip renders on any group yet: the rigs run pre-0.2.0 builds whose akka check carries no data object. This is the designed graceful degradation, and it confirms Phase 4 must redeploy before checking design 9 item 2.", + "SUSPECTED RIG SPLIT-BRAIN: scadabridge-central-a AND central-b both return /health/active 200 with 'oldest Up member (singleton host)'. Two nodes cannot both be the oldest of one cluster. Verify during Phase 4 after redeploy \u2014 this is exactly the condition the dashboard exists to surface.", + "PRODUCT FIX during acceptance: Program.cs now calls builder.WebHost.UseStaticWebAssets() unconditionally. The host only calls it when the environment is literally 'Development', so running the build output under any other name (checking a staging registry locally) left the Theme RCL's _content assets 404 and the page rendered as a blank white screen with its markup fully present. No-op once published, so the container was never affected.", + "RIG DEFECT (pre-existing, both products): self-first Akka seed lists mean a pair booted cold self-forms two 1-member clusters that never join. Observed fleet-wide on ScadaBridge after a --force-recreate: every node named ITSELF leader and every node answered /health/active 200. Restarting ONE node of a pair makes it join the survivor and the pair converges to one Leader + one Active + one Standby. This is the user's own open backlog item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted). The dashboard surfaced it fleet-wide in one screen, which is what it is for.", + "Leader and Active legitimately diverge after a restart: Akka leader is lowest-address, ScadaBridge's active-node check is oldest Up member. site-c-a shows LEADER + STANDBY while site-c-b shows ACTIVE. Same divergence CLAUDE.md documents for OtOpcUa's Primary vs RoleLeader.", + "OUTSTANDING: the OtOpcUa docker-dev rig still runs a pre-0.2.0 build. Redeploying it needs a decision because the repo is checked out on feat/mesh-phase6, which carries 4 commits and an untracked Phase 7 plan beyond the base of feat/health-0.2.0-bump.", + "PRODUCT BUG found by the live rig: LeaderResolver and InstanceCard.ClusterDataLine both keyed the cluster view on the check NAME 'akka-cluster'. ScadaBridge registers the shared AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as 'akka'. The result was a silent miss \u2014 OtOpcUa groups rendered with no leader chip and no evidence line, which is indistinguishable from a group that simply has not reported yet. Both now locate the entry by the DATA KEY ('leader'), which is the part of the contract the shared check owns. Regression tests cover akka-cluster / akka / an arbitrary future name, plus an entry that publishes data WITHOUT a leader (ScadaBridge's localdb check) so scanning cannot pick the wrong entry. Unit tests and review had both missed it because every fixture used ScadaBridge's name.", + "OTOPCUA CONTRACT GAP (not a dashboard bug, worth filing): /health/active does not answer 'am I the active node'. On the standby admin (central-2) the admin-leader check returns Degraded ('Role admin member but not leader'), and ASP.NET maps Degraded to 200 \u2014 so a standby looks Active. On driver-only nodes it returns Healthy with the description 'Active for role admin (or not a role member)', i.e. it passes vacuously for every node that is not an admin at all. Meanwhile CLAUDE.md documents the real election as the oldest Up driver member per Cluster (IsDriverPrimary, ServiceLevel 250 vs 240), which this tier never consults. For the tier to mean what MapZbHealth's status mapping implies, a non-active node must return Unhealthy (503) \u2014 which is exactly what ScadaBridge's SitePairActiveNodeHealthCheck does, and why ScadaBridge reads correctly. FILED as lmxopcua#494 (https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494), with the traefik-dynamic.yml LB-healthcheck impact flagged: docker-dev uses /health/active as its load-balancer probe, so making a standby return 503 would drop it from the UI pool.", + "OtOpcUa rig rebuilt from a new branch feat/mesh-phase6-health-0.2.0 = feat/mesh-phase6 + a cherry-pick of the 3-line Health 0.2.0 bump (2e515c34). Checking out feat/health-0.2.0-bump directly would have regressed the rig past four commits of in-progress Phase 6/7 work. All three meshes formed cleanly (2 members, 0 unreachable, both members agreeing on one leader) \u2014 OtOpcUa's per-pair self-first seeding did NOT hit the cold-boot self-form race that ScadaBridge did.", + "TRAEFIK LEADER-PINNING FIXED (ZB.MOM.WW.Health 0.2.1, published + restore-verified; OtOpcUa bumped at 8dd9da7d). Root cause was in the shared library: ActiveNodeDecision.Evaluate returned Degraded for role-member-but-not-leader, and MapZbHealth maps Degraded to 200 \u2014 so /health/active answered 200 on every node and OtOpcUa's Traefik kept the standby in the admin-UI pool. The shared spec had always said the tier 'Fails (503) on a standby or role-member-but-not-leader node', and OtOpcUa's own docs said '200 only on the Admin-role leader; 503 elsewhere'; the docs were right and the code was wrong. Live-verified: central-1 200/traefik UP, central-2 503/traefik DOWN; the README failover drill (which had never exercised anything) flipped Traefik to central-2 within 20s of stopping central-1 with the UI answering 200 continuously across the swap, and central-1 reclaimed the role-leader on restart. Blast radius one consumer \u2014 ScadaBridge uses its own OldestNodeActiveHealthCheck, mxgw/HG have no Akka.", + "RIG OPERATIONS LANDMINE: the docker-dev migrator (dotnet ef database update) HANGS under x86 emulation \u2014 22 minutes at 0.00% CPU with no output, and all six node containers sit in Created because they depend_on service_completed_successfully. Worked around by stopping the migrator and docker-start-ing the nodes directly, which is safe when the change is a package-version bump with no EF model change. Worth knowing before assuming a rebuild is merely slow.", + "lmxopcua#494 CLOSED. The site-node half needed a local ClusterPrimaryHealthCheck: this node is active iff it is the OLDEST Up member carrying its own active role (admin when present, else driver). IMPORTANT SELF-CORRECTION \u2014 the Health 0.2.1 fix alone was NOT sufficient. It made the tier a real 503 but still selected by RoleLeader (lowest address), and on the rebuilt rig the orderings diverge live: akka leader = central-1, oldest admin member = central-2 (which hosts the singletons). 0.2.1 alone would have pinned Traefik to central-1, the node NOT hosting the work \u2014 the exact failure RedundancyStateActor.SelectDriverPrimary was written to avoid, reappearing in the health tier. Only the live rig exposed it. Follow-up not filed: two of four family apps now need 'oldest Up member of role X' and both had to avoid the shared ActiveNodeHealthCheck to get it (ScadaBridge's OldestNodeActiveHealthCheck, OtOpcUa's ClusterPrimaryHealthCheck) \u2014 a candidate for promotion into ZB.MOM.WW.Health.Akka later.", + "RIG BUILD LANDMINE: all six docker-dev node services share ONE image (otopcua-host:dev), so `docker compose build` with six service names builds the same image six times in parallel. Under x86 emulation that thrashed for 46+ minutes on `dotnet restore` with no output and looked wedged. Building a single service took 29 SECONDS for the same restore. Build one service, then recreate all six containers." + ], + "phase4_results": { + "1_all_render_single_active_per_pair": "PASS for both products. ScadaBridge: all four pairs one Active + one Standby. OtOpcUa: after lmxopcua#494 was fully closed (ClusterPrimaryHealthCheck, commit 45504861), exactly one 200 per mesh \u2014 MAIN central-2, SITE-A site-a-1, SITE-B site-b-1 \u2014 with the partner 503. Dashboard renders one Active + one Standby for all seven cluster groups.", + "2_leader_chip_and_split_brain": "PASS for BOTH products after the rig redeploy and the LeaderResolver fix. All seven cluster groups (ScadaBridge central/site-a/b/c, OtOpcUa MAIN/SITE-A/SITE-B) render a Leader chip with zero split-brain flags. Verified earlier that the chip MOVES when the leader is stopped and that the split-brain flag appears while a pair is self-formed and clears once it genuinely joins.", + "3_kill_and_restart": "PASS. Stopped scadabridge-site-c-a 12:01:27; Unreachable by 12:02:33 (two-strike damping = 2 failing sweeps). Restarted 12:04:01; Up again by 12:04:16, i.e. on the first good poll (recovery is deliberately asymmetric)", + "4_degraded_with_failing_entry": "PASS. Canned Degraded and Unhealthy endpoints render as Degraded/Down with the failing entry and its description visible, and check-dot ok/warn/bad classes per check", + "5_cache_first": "PASS. Five consecutive page loads in 0.00-0.02s, all showing the identical 'last sweep 12:17:44' header \u2014 the render never triggers a poll (an inline sweep could not finish in 10ms with two unreachable targets on 3s/10s timeouts)", + "6_pause_and_stuck_poller": "PASS both halves. Pause -> both cards STALE within the 10s window, KPI stale=2, 'Stale since'; Resume -> clean recovery. Separately, SIGSTOPping the endpoints while auto-refresh stayed RUNNING froze the poller mid-sweep and the cards aged into STALE on their own \u2014 staleness is computed at render, so a stuck poller needs no explicit stall detection", + "7_management_links_no_login": "PASS. 'Open CentralUI' opened http://localhost:9001 in a new tab (target=_blank rel=noopener); the dashboard itself never presented or redirected to a login", + "8_malformed_registry": "PASS both branches. No registry -> ConfigPreflight names Overview:Applications:0:Name and :Instances:0:BaseUrl. Stale window inside the poll interval -> validator names the keys and the consequence", + "9_health_and_metrics": "PASS. /health/ready, /health/active, /healthz, /metrics all 200; 16 gauge series with application/node/group labels" + } + } +} \ No newline at end of file