Merge branch 'master' into fix/r3-02-stale-good-family
# Conflicts: # src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs # tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Harness/V3ArtifactSeeder.cs
This commit is contained in:
Joseph Doherty
@@ -79,12 +79,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
|---|---|---|---|
|
||||
| `Username` | string | `apikey` | The fixed username a key-bearing client sends in its OPC UA UserName token. Carries no information — the key identifies the principal — but is checked rather than ignored so a typo is an explicit denial. |
|
||||
| `Pepper` | string | `""` | Server-side pepper for the stored key hashes. **Supply via `Security__ApiKeys__Pepper` or `${secret:}` — never commit it.** No default on purpose: an empty pepper still hashes "successfully", so `ApiKeySecret.ComputeHash` throws on one rather than producing a deployment with none of the protection it believes it has. Rotating it invalidates every existing key. |
|
||||
| `StoreTimeoutMs` | int | `2000` | Hard per-call deadline on the API-key store lookup that runs inline on session activation (arch-review 03/S18). Exceeding it **denies** the activation, logs a distinct Warning and meters `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. `≤ 0` disables the deadline — do not, on a node with a ConfigDb: the SDK holds a `SessionManager`-wide lock across that lookup, so an unanswered query stalls **every** session activation on the server. Applies only to `ConfigDbApiKeyStore`; the in-memory artifact store on a driver-only node is not wrapped. |
|
||||
| `StoreCacheSeconds` | int | `5` | TTL of the read-through cache in front of that store, so a burst of activations costs one round trip. `0` disables it (a store read per activation). Bounds how long an `Enabled` revocation authored in the ConfigDb takes to reach a **new** activation; it has no effect on already-open sessions (those re-check the artifact snapshot) and cannot delay an `ExpiresUtc` — a cached hit that has since expired is re-read rather than served. |
|
||||
|
||||
**`Security:Acl`** key fields (`AclOptions`):
|
||||
|
||||
| Key | Type | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `Enabled` | bool | `false` | `true` makes scoped `NodeAcl` grants decide **write**, **alarm-command** and **HistoryRead** authorization, *replacing* the server-wide `WriteOperate` / `AlarmAck` role checks. `false` keeps those role checks and leaves HistoryRead ungated — byte-for-byte the pre-stage-5 behaviour. Read, Browse and Subscribe are ungated either way. |
|
||||
| `RevalidationIntervalSeconds` | int | `60` | How often live monitored items are re-authorized against the clock, in addition to on every deployment (arch-review 03/S19). A key's `ExpiresUtc` passes with the clock, so without this an expired key kept feeding a live subscription until a deployment happened to land — indefinitely on a fleet deployed once and left running. `0` disables the timer (deploy-triggered only). **Inert while `Enabled` is false**, since the pass early-returns when enforcement is off. An affected session is silenced with `BadUserAccessDenied`, **not closed**, and recovers on re-activation. |
|
||||
|
||||
> **Flipping `Security:Acl:Enabled` is a config change, not a redeploy** — both paths are wired in both modes (the node builds its permission trie on every deploy regardless), the same dark-switch discipline as [`MeshTransport`](#meshtransport), [`ConfigSource`](#configsource) and [`Telemetry`](#telemetry). It defaults off because the first deployment carrying the code would otherwise start denying every client that has no key, **including ScadaBridge's Data Connection Layer, which consumes this address space from another repo.** Enforcement fails closed on every unresolvable case, so a wiring or config mistake presents as *everything denied* — check the `OpcUaPublish: data-plane authority generation N installed` line for a non-zero grant count before debugging an individual rule. See [`security.md` § Node ACL enforcement](security.md#node-acl-enforcement-securityaclenabled).
|
||||
|
||||
@@ -190,6 +193,8 @@ On the docker-dev rig every node carries these keys already; the rig's compose d
|
||||
|
||||
See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.
|
||||
|
||||
**On an admin-role node, `NodeCommandService` and `CentralCommandService` are `AllowAnonymous` by design** — the AdminUI cookie `FallbackPolicy` would otherwise 401/redirect a correctly-PSK'd call before the interceptor above ever ran (archreview 03/S14). See [`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).
|
||||
|
||||
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
|
||||
|
||||
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
|
||||
@@ -217,6 +222,8 @@ See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-t
|
||||
|
||||
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
|
||||
|
||||
**`DeploymentArtifactService` is `AllowAnonymous` by design** — same reasoning as `NodeCommandService`/`CentralCommandService` above: the AdminUI cookie `FallbackPolicy` on an admin-role node would otherwise block a correctly-keyed fetch before `ConfigServeAuthInterceptor`'s verdict matters. See [`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).
|
||||
|
||||
#### Artifact compression (2026-08-15)
|
||||
|
||||
The artifact blob is **gzip-compressed, inside the hashed bytes**. `ConfigComposer.Pack` gzips the
|
||||
|
||||
+109
-1
@@ -164,6 +164,93 @@ mostly reusable — what was always missing is the trigger.
|
||||
|
||||
---
|
||||
|
||||
## What a (re)spawn re-provisions
|
||||
|
||||
A driver child can come into existence by **three** routes, and until 2026-08-20 only the first of them
|
||||
handed the child everything it needs to be a working driver. That is archreview **03/S16** (and its worse
|
||||
half, **03/S6**): a child that exists without its desired subscription set is a *dead* driver — it
|
||||
connects, reports `Connected`, publishes nothing and raises no alarms, and the #488 periodic reconcile
|
||||
cannot rescue it (it early-returns on an empty desired set). The fix makes the desired set **part of what
|
||||
it means to have a child** rather than a later step in the apply pipeline.
|
||||
|
||||
**`DriverHostActor._desiredByDriver` is the single authority.** It is a per-driver
|
||||
`(ValueRefs, AlarmRefs)` pair, cleared and repopulated **whole** on every apply in the same pass that
|
||||
rebuilds the routing maps — keyed over the union of the value-ref map, the alarm-ref map and the running
|
||||
`_children`, so a driver with no authored tags gets an explicit **empty** set rather than being absent.
|
||||
Everything that hands a child a `SetDesiredSubscriptions` goes through the one helper
|
||||
`SendDesiredSubscriptions(actor, driverId)`. (An empty set is meaningful, not a no-op to skip: the child
|
||||
routes it to `Unsubscribe`, dropping a stale handle.)
|
||||
|
||||
| Route | Trigger | What re-provisions it |
|
||||
|---|---|---|
|
||||
| **Deploy reconcile** | `ApplyDriverSpecs` → `SpawnChild` | `SpawnChild` sends `InitializeRequested` then the desired set; the apply's bulk push then sends the (freshly rebuilt) set to every child. |
|
||||
| **Operator Restart** | `/hosts` → Restart → `HandleRestartDriver` | Stops the child, respawns from the cached spec — and `SpawnChild` sends the desired set, because it always does. **This is the 03/S16 fix**: there is no bulk push on this path, and before it the respawned child subscribed to nothing until the next deployment. |
|
||||
| **Supervision restart** | a child throws ⇒ Akka's **default decider** ⇒ `Restart` | `DriverInstanceActor.PostRestart` Tells the host `ReprovisionRequested`; `HandleReprovisionRequested` logs a Warning naming the crash type, re-sends `InitializeRequested` (non-stub children) and the desired set. |
|
||||
| **Force Reconnect** | `/hosts` → Force Reconnect → `HandleReconnectDriver` | **Nothing — verified.** It Tells `ForceReconnect` to the *same* child actor, so the instance and its `_desiredRefs`/`_desiredAlarmRefs` survive and the reconnect's Connected entry re-applies them itself. Pinned by `Force_reconnect_keeps_the_desired_set_and_re_subscribes` so a future edit to the restart paths cannot break the one that was already correct. |
|
||||
|
||||
Things to know before touching any of this:
|
||||
|
||||
- ⚠️ **The supervision route is the invisible one, and it was the worst.** There is **no**
|
||||
`SupervisorStrategy` anywhere under `src/Server`, so a child that throws gets Akka's default `Restart`.
|
||||
A restart recreates the actor object around the **same** `IDriver` — the Props closure captured it —
|
||||
and its `PreRestart` runs `PostStop`, which calls `IDriver.ShutdownAsync`. It does **not** run
|
||||
`SpawnChild`, and the child's `IActorRef` is unchanged, so the host had no notification a new
|
||||
incarnation existed at all. Net effect before the hook: the child sat in `Connecting` forever around a
|
||||
shut-down driver, with its desired set gone, and nothing logged the non-recovery.
|
||||
- **Replacing the default decider is deliberately out of scope.** A `Stop` directive would silently
|
||||
remove a driver; a bounded retry count would silently stop one after N crashes. Both change failure
|
||||
semantics for every driver in the fleet and deserve their own plan and live gate. The reprovision hook
|
||||
makes the *current* infinite-restart behaviour correct instead of dead.
|
||||
- ⚠️ **…which is exactly why the reprovision is RATE-LIMITED.** The default decider restarts a faulting
|
||||
child **immediately and without limit**, and the hook now makes every one of those restarts do
|
||||
`ShutdownAsync` + `InitializeAsync` + `SubscribeAsync` — so a driver that faults on every value would
|
||||
hammer its device as fast as it can crash. `DriverHostActor` keeps a per-driver last-reprovision stamp
|
||||
and holds anything inside `DefaultReprovisionBackoff` (10 s, matched to
|
||||
`DriverInstanceActor.DefaultReconnectInterval` because it drives the same connect work). The held
|
||||
request is **deferred, not dropped** — one `StartSingleTimer` per driver, so a burst collapses to a
|
||||
single pending reprovision — because dropping it would leave a child whose *final* crash fell inside
|
||||
the window un-provisioned forever, which is the 03/S16 failure this hook exists to close. Each
|
||||
deferral logs a Warning naming the crash type, so a crash loop is visible rather than merely quiet.
|
||||
- **A reprovision is routed by INCARNATION, not by driver id.** `ReprovisionRequested` is delivered
|
||||
asynchronously, so one can still be in flight from an old incarnation after `HandleRestartDriver`
|
||||
stopped that child and spawned a replacement under the same id. Delivered to the new child — which is
|
||||
in `Connecting`, where `InitializeRequested` *is* handled — it would start a second, concurrent
|
||||
`InitializeAsync`, racing the dispose-before-overwrite that #527's teardown exists to serialise
|
||||
(`ModbusDriver._transport` / `MqttDriver._connection` each leak a transport on that race). The handler
|
||||
therefore checks `Sender` against the child it currently holds and drops anything else; the deferred
|
||||
path carries the original ref so the same check still applies when the timer re-delivers it.
|
||||
- **Ordering on a respawning deploy, and its one accepted cost.** `ApplyDriverSpecs` (which spawns) runs
|
||||
before the pass that rebuilds `_desiredByDriver`, so a child respawned by a deploy — raw tags are folded
|
||||
into the merged `DriverConfig`, so a tag-set edit changes the spec and trips the #516 config respawn —
|
||||
is first handed the **previous** apply's set, then the new one by the bulk push. Akka's per-pair
|
||||
ordering guarantees it **converges**; it does not guarantee the transient is inert, because the child
|
||||
runs concurrently with the rest of the apply handler and a fast-initialising driver can reach
|
||||
`Connected` and subscribe the stale set first. The cost is one extra subscribe round-trip
|
||||
(`HandleSubscribeAsync` drops the prior subscription before establishing the new one); a real device
|
||||
driver, whose initialise opens a socket, rarely wins that race. Withholding the spawn-time send to avoid
|
||||
it would simply restore 03/S16, since the Restart and supervision routes have no bulk push coming.
|
||||
- ⚠️ **Convergence is not free on the ALARM side, and "a subscription already exists" is the wrong guard
|
||||
there.** The value path converges by construction — `StoreDesiredSubscriptions` is a full replace and
|
||||
`HandleSubscribeAsync` drops the prior subscription before establishing the new one — but
|
||||
`SubscribeDesiredAlarms` / `HandleSubscribeAlarmsAsync` used to gate purely on
|
||||
`_alarmSubscriptionHandle is not null`, so once a handle existed the alarm set **latched for the life of
|
||||
the connection**. Combined with the transient above, a deploy widening a driver's alarm refs `{X}` →
|
||||
`{X, Y}` could leave a fast-initialising driver subscribed to `{X}` forever, and the drivers that filter
|
||||
their feed on the subscribed `sourceNodeIds` (OPC UA Client, AB CIP, FOCAS) would simply never deliver
|
||||
the new alarm — silently. The guard is now "the desired refs differ from the ones the live handle was
|
||||
established with" (`_subscribedAlarmRefs`), and a changed set **releases the old subscription before
|
||||
re-subscribing** so a filtering driver does not accumulate two overlapping ones. There is still
|
||||
deliberately no unsubscribe-on-**empty**: a removed alarm tag's condition node is torn down on the
|
||||
address-space rebuild and `ForwardNativeAlarm` drops any transition whose `ConditionId` no longer maps.
|
||||
- **Restart tests must assert a FUNCTION, never existence.** `RestartDriver_respawns_the_child_without_an_actor_name_collision`
|
||||
asserts only that a child exists after a restart, and it passed green against the dead driver 03/S16
|
||||
describes. It is kept — its own subject, the `InvalidActorNameException`, is real — but the load-bearing
|
||||
suite is `DriverHostActorRestartTests`, which asserts that the new incarnation *subscribed* and
|
||||
*re-initialised*. Its file remarks also record the supervision crash seam and why the obvious
|
||||
alternatives (a throwing health publisher, a throwing `SubscribeAsync`, `Kill.Instance`) do not work.
|
||||
|
||||
---
|
||||
|
||||
## IDriverHealthPublisher — health pub/sub sink
|
||||
|
||||
[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs)
|
||||
@@ -267,7 +354,8 @@ The `Connected`-state health tick now also evaluates a watchdog, and self-sends
|
||||
|
||||
1. the driver reports a state other than `Healthy`, **and**
|
||||
2. `DriverHealth.LastSuccessfulRead` has not advanced for `DefaultCommsLossTimeout` (90 s = three ticks),
|
||||
measured from the Connected entry when the driver has never reported a successful read.
|
||||
measured from **the later of that read stamp and the current Connected entry** (see "the grace is per
|
||||
connection" below).
|
||||
|
||||
**The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule.** `Degraded` means
|
||||
*connected but some data is bad* — the enum says so. Several drivers publish health **per tag inside their
|
||||
@@ -281,6 +369,26 @@ still needed as the second conjunct, because a `Healthy` driver with nothing sub
|
||||
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
|
||||
`DriverInstanceActorCommsLossWatchdogTests`, each of which goes red when the *other* conjunct is deleted.
|
||||
|
||||
**The grace is per CONNECTION — the baseline is `max(LastSuccessfulRead, connectedSince)`, not
|
||||
`LastSuccessfulRead ?? connectedSince`** (archreview 03/S17, fixed 2026-08-20). The two spellings differ
|
||||
for exactly the case that matters, and the `??` one collapsed the grace to a single health tick. Because
|
||||
`LastSuccessfulRead` is *preserved* across `Degraded`/`Faulted` writes — that preservation is the design
|
||||
described above — a device that accepts a TCP connection but never serves data (a half-broken PLC, a wrong
|
||||
unit id, a firewalled read path) reconnects carrying a read stamp frozen from before the outage, already
|
||||
far older than the threshold. Measuring from that stamp alone re-trips on the **first** tick of every new
|
||||
connection, so the reconnect cycle ran at health-tick cadence (~30 s in production) rather than at the
|
||||
designed 90 s. Taking the *later* of the two gives each connection its own full window, which is what
|
||||
re-stamping `connectedSince` on every Connected entry was always for; `??` only ever helped a driver that
|
||||
had never read at all, and the max still covers that case.
|
||||
|
||||
What this does **not** change: the conjunction, the disable-at-zero hatch, and a driver that stops reading
|
||||
*within* a stable connection (its read stamp is the later term, so it still trips on schedule measured
|
||||
from that read). And it **paces** the half-broken-device cycle rather than ending it — expect at most one
|
||||
reconnect attempt per threshold per connection, not silence. Pinned by
|
||||
`A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again` (the delay) and
|
||||
`A_half_broken_device_is_still_declared_lost_once_the_grace_elapses` (the anti-vacuity half — without it,
|
||||
a "fix" that simply disabled the watchdog after the first connection would look correct).
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
|
||||
|
||||
@@ -190,6 +190,14 @@ transports. ScadaBridge itself has since closed that gap with this identical
|
||||
interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a
|
||||
later hardening pass. See the design doc's superseded note for the full history.
|
||||
|
||||
**On a fused `admin,driver` node, `TelemetryStreamService` and `NodeCommandService` are `AllowAnonymous`
|
||||
by design** — a *different* gate from the interceptor above. `TelemetryStreamAuthInterceptor` is the PSK
|
||||
check; on an admin-role node the AdminUI cookie `FallbackPolicy` sits in front of it and would 401/redirect
|
||||
a correctly-keyed call before the interceptor's verdict matters, because it runs in the authorization
|
||||
middleware, strictly *before* the interceptor executes (archreview 03/S14). A driver-only node never
|
||||
registers that middleware at all, so it is structurally unaffected either way. See
|
||||
[`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).
|
||||
|
||||
## Upgrading an existing deployment: populate `ClusterNode.GrpcPort` first (#493)
|
||||
|
||||
`GrpcPort` is a **nullable column added by Phase 5**, and nothing backfills it onto rows that
|
||||
|
||||
+151
-16
@@ -177,16 +177,59 @@ When no authenticator is supplied, `OpcUaApplicationHost` falls back to `NullOpc
|
||||
the Host wires the real `ApiKeyOpcUaUserAuthenticator` as a singleton in `Program.cs`. Which
|
||||
**`IApiKeyStore`** backs it depends on the node's role:
|
||||
|
||||
| Node | Store | Revocation takes effect |
|
||||
|---|---|---|
|
||||
| `admin` (including fused `admin,driver`) | `ConfigDbApiKeyStore` — reads the config database per request | On the **next request** |
|
||||
| `driver`-only | `ArtifactApiKeyStore` — reads the keys carried by the deployment artifact, held in memory | On the **next deployment** |
|
||||
| neither role | `NullOpcUaUserAuthenticator` — denies every UserName token | n/a |
|
||||
| Node | Store | An `Enabled = 0` revocation reaches a **new activation** | …and an **already-open session** |
|
||||
|---|---|---|---|
|
||||
| `admin` (including fused `admin,driver`) | `ConfigDbApiKeyStore` — reads the config database per activation, wrapped in `BoundedApiKeyStore` | On the **next activation**, bounded by `Security:ApiKeys:StoreCacheSeconds` (default 5 s) | On the **next deployment** |
|
||||
| `driver`-only | `ArtifactApiKeyStore` — reads the keys carried by the deployment artifact, held in memory | On the **next deployment** | On the **next deployment** |
|
||||
| neither role | `NullOpcUaUserAuthenticator` — denies every UserName token | n/a | n/a |
|
||||
|
||||
⚠️ **The last column is the correction, and it used to be missing.** This table once said a ConfigDb node
|
||||
revokes "on the next request", which is true only of a **new session activation**. An operation on a
|
||||
session that is *already open* does not re-read `IApiKeyStore` at all — `DataPlaneAuthorizer.Evaluate`
|
||||
re-checks the key against the in-memory **artifact snapshot** (`DataPlaneAuthority.FindUsableKey`), which
|
||||
`OpcUaPublishActor` installs on every apply. So clearing `Enabled` in the ConfigDb stops the *next*
|
||||
connect on an admin node and nothing else until a deployment lands, on **both** node kinds.
|
||||
|
||||
**Expiry is the exception, and it is immediate on both.** `FindUsableKey` compares `ExpiresUtc` against
|
||||
the authorizer's clock on every call, so an `ExpiresUtc` that has already passed denies at once on an
|
||||
open session — and since arch-review 03/S19 it reaches a live **subscription** too, on a timer (see
|
||||
[Data-Plane Authorization](#data-plane-authorization)).
|
||||
|
||||
The deny-all is registered **explicitly**, not left absent: `OtOpcUaServerHostedService` takes
|
||||
`IOpcUaUserAuthenticator` as a required constructor parameter, so an unregistered seam fails DI and the
|
||||
node does not boot at all.
|
||||
|
||||
### Bounding the key store (`StoreTimeoutMs` / `StoreCacheSeconds`)
|
||||
|
||||
The SDK raises impersonation **synchronously, holding a `SessionManager`-wide event lock**, so the key
|
||||
lookup does not stall one activation — it serializes **every** session activation on the server,
|
||||
Anonymous and X.509 included. On a driver-only node that is harmless (the artifact store is two
|
||||
dictionary probes). On an admin or fused node it is two EF round trips to central SQL, so a ConfigDb that
|
||||
is *slow or hung* rather than *refusing* could hold that lock indefinitely. `BoundedApiKeyStore`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/BoundedApiKeyStore.cs`) wraps the ConfigDb store — and only it —
|
||||
with two `Security:ApiKeys` knobs:
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `StoreTimeoutMs` | `2000` | Hard per-call deadline on the store. Exceeding it **denies** the activation. `≤ 0` disables the deadline (the pre-R3-05 unbounded behaviour — do not do this on an admin node). |
|
||||
| `StoreCacheSeconds` | `5` | TTL of a read-through cache in front of the store, so a burst of activations costs one round trip. `0` disables it (a read per activation). |
|
||||
|
||||
- **A timeout denies; it never admits.** The deadline surfaces as an `OperationCanceledException` into
|
||||
`ApiKeyOpcUaUserAuthenticator`'s existing catch-all, which returns the same opaque denial as a bad
|
||||
credential. There is no second denial path to get wrong, and the decorator can only ever return what
|
||||
the inner store returned or throw.
|
||||
- **What a timeout looks like:** a distinct Warning — *"ApiKeyStore: the key store did not answer within
|
||||
N ms — denying this activation. Every OPC UA session activation on this server is serialized behind
|
||||
this call; check ConfigDb health"* — plus
|
||||
`otopcua.opcua.apikey_store.bounded{outcome=timeout}`. It is deliberately **not** the ordinary
|
||||
*"no enabled, unexpired key with id X"* line, which would send an operator hunting for a key problem
|
||||
that does not exist.
|
||||
- **What to watch:** `otopcua.opcua.apikey_store.bounded{outcome=cache_hit}`. A busy fused central with no
|
||||
`cache_hit` stream means `StoreCacheSeconds` is 0.
|
||||
- **Faults and timeouts are never cached** — caching a deny would extend an outage past the outage. Only
|
||||
a completed lookup (a hit or an authoritative miss) is stored, and a cached hit whose `ExpiresUtc` has
|
||||
since passed is re-read rather than served, so the TTL can never resurrect an expired key.
|
||||
|
||||
### LDAP bind flow (`OtOpcUaLdapAuthService`)
|
||||
|
||||
LDAP is configured under the `Security:Ldap` section (bound to `LdapOptions`, `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs`, `SectionName = "Security:Ldap"`). The app authenticator is `OtOpcUaLdapAuthService` — a thin wrapper around the shared `ZB.MOM.WW.Auth.Ldap` directory client that adds two app-only concerns the shared library deliberately does not model: the `Enabled` master switch and `DevStubMode`. Since #520 it serves the **Admin UI cookie login (`/auth/login`) only** — `LdapOpcUaUserAuthenticator` is deleted and the OPC UA UserName path authenticates API keys instead (see [API-key identity](#api-key-identity-opc-ua-data-plane)).
|
||||
@@ -411,17 +454,36 @@ Behaviours worth knowing before enabling it:
|
||||
- **A deploy takes effect on sessions that are already open.** The gate stamps each decision with the
|
||||
current authority generation rather than the one the session activated under, so an ACL edit lands
|
||||
without bouncing clients — in both directions, widening and narrowing.
|
||||
- **Revocation also reaches an open session.** Every decision re-checks the authenticating key against
|
||||
the current snapshot, so a key disabled or expired by a later deployment stops working immediately
|
||||
rather than at the client's next reconnect. On a driver-only node the artifact is the only revocation
|
||||
channel there is, so without this a revoked credential could keep its authority indefinitely.
|
||||
- **Revocation reaches a live subscription too** (#520 stage 6). Every other operation re-asks on each
|
||||
request; a subscription is authorized once and then streams, so it needed a push. A deployment
|
||||
re-authorizes every live monitored item: one whose grant has gone is queued a `BadUserAccessDenied`
|
||||
and then detached from its node's value fan-out, and a later deployment that restores the grant
|
||||
re-attaches it. This matters more than it sounds — subscriptions are how machine consumers actually
|
||||
read this server, so without it revoking a grant would close the operations nobody uses and leave the
|
||||
feed running.
|
||||
- **Revocation also reaches an open session — but read the two axes separately.** Every decision
|
||||
re-checks the authenticating key against the current snapshot, so an open session is re-authorized on
|
||||
every operation. What that check can *see* differs by axis:
|
||||
- **The deployment axis** (a key's `Enabled` cleared, a grant edited or removed). The snapshot only
|
||||
changes when an artifact is applied, so this lands **on the next deployment**, on both node kinds —
|
||||
including on a fused admin node, whose ConfigDb store is consulted at *activation* only. On a
|
||||
driver-only node the artifact is the only revocation channel there is.
|
||||
- **The clock axis** (a key's `ExpiresUtc` passing). `FindUsableKey` compares it against the
|
||||
authorizer's clock on every call, so this lands **immediately**, with no deployment.
|
||||
- **Revocation reaches a live subscription too** (#520 stage 6, extended by arch-review 03/S19). Every
|
||||
other operation re-asks on each request; a subscription is authorized once and then streams, so it
|
||||
needed a push. A denied item is queued a `BadUserAccessDenied` and then detached from its node's value
|
||||
fan-out; restoring its authority re-attaches it. There are **two triggers**:
|
||||
- a **deployment**, which re-authorizes every live monitored item (the deployment axis);
|
||||
- the host's **low-frequency timer**, `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0`
|
||||
disables it, inert while `Security:Acl:Enabled` is false). Without it a key that expired by the
|
||||
clock kept streaming until a deployment happened to land — **indefinitely** on a fleet that has been
|
||||
deployed once and left running, which is exactly the shape a production fleet settles into.
|
||||
|
||||
⚠️ **An expired-key session is silenced, not disconnected.** Its items go `BadUserAccessDenied` and the
|
||||
session stays open, deliberately: a clock-driven denial must look to a client exactly like a
|
||||
deploy-driven one, OPC UA has no honest "expire this session" verb (deleting it server-side gives the
|
||||
client a transport failure rather than an authorization status), and every other operation on that
|
||||
session already denies. **A client that re-activates with a renewed key recovers on the next tick with
|
||||
no reconnect** — closing the session instead would force a full reconnect-and-resubscribe on every
|
||||
machine consumer in the fleet at once when a shared key expired.
|
||||
|
||||
⚠️ **Event/alarm monitored items are revalidated by neither trigger.** The live-item map is fed only
|
||||
from data-change items, so an alarm subscription is gated at creation and never again. Pre-existing —
|
||||
the deployment trigger has the same hole — and tracked as a residual of arch-review 03/S19.
|
||||
- **An ancestor of a granted node is browsable**, or no narrow grant would be reachable: a client
|
||||
arrives at a deep node by browsing its ancestors, and the trie otherwise inherits downward only. Note
|
||||
what this necessarily discloses — the *names* of the folders on the path to a granted node are visible
|
||||
@@ -846,6 +908,79 @@ Responses: `202 Accepted` (`{ outcome, deploymentId, revisionHash }`) when a dep
|
||||
|
||||
---
|
||||
|
||||
## Mesh gRPC ingress is `AllowAnonymous` by design
|
||||
|
||||
Archreview 03/S14 (Critical, closed R3-01). On an admin-role node `AddOtOpcUaAuth` installs a global
|
||||
`FallbackPolicy = RequireAuthenticatedUser` bound to the cookie scheme (the same fallback that protects
|
||||
every AdminUI page above). ASP.NET Core's authorization middleware applies that fallback to **every
|
||||
endpoint that carries no authorization metadata of its own** — gRPC endpoints included. The five mesh
|
||||
gRPC endpoints (config-serve artifact fetch, central command ack, telemetry stream, node command, LocalDb
|
||||
sync) authenticate with a per-cluster/per-node **PSK bearer inside a gRPC interceptor**, and that
|
||||
interceptor runs *inside* the endpoint invocation — i.e. strictly *after* the authorization middleware.
|
||||
Without an explicit `.AllowAnonymous()` opt-out, a correctly-keyed node call is challenged by the cookie
|
||||
handler before the interceptor's verdict ever matters: on a login-enabled admin node this is a fleet-wide
|
||||
deploy-plane outage (artifact fetches fail, deploy acks are dropped, deployments seal `TimedOut`), and it
|
||||
went unnoticed for seven phases of prior live gates because the docker-dev rig always ran
|
||||
`Security__Auth__DisableLogin=true` (an always-succeeding handler registered *under the cookie scheme
|
||||
name*, so the fallback trivially passes) and every prior mesh-gRPC test built its own auth-free pipeline
|
||||
by hand.
|
||||
|
||||
**Why `AllowAnonymous` is the correct fix, not a hole.** All five interceptors are fail-closed: an unset
|
||||
key rejects every call, and the presented token is compared with `CryptographicOperations.FixedTimeEquals`.
|
||||
`Program.cs:574` states the design contract directly — "the interceptor is the ONLY inbound auth on any
|
||||
endpoint (the libraries verify nothing)" — which was already true of the gRPC pipeline; the missing opt-out
|
||||
made it false of the HTTP pipeline in front of it. The pattern itself predates this fix: static assets
|
||||
(`MapStaticAssets().AllowAnonymous()`), the metrics endpoint (`MapZbMetrics().AllowAnonymous()`), and the
|
||||
headless deploy API above are all `AllowAnonymous` for the identical reason — each is gated by something
|
||||
other than the cookie, and the fallback would otherwise 401/redirect it regardless.
|
||||
|
||||
| Endpoint | Role gate | Affected without `AllowAnonymous`, on a login-enabled node |
|
||||
|---|---|---|
|
||||
| `DeploymentArtifactService.Fetch` | `hasAdmin && ConfigServe:GrpcListenPort > 0` | Always — every `FetchAndCache` node's artifact fetch. |
|
||||
| `CentralCommandService.ReportApply` | same | Always — every node's deploy ack, from every cluster. |
|
||||
| `TelemetryStreamService.Subscribe` | `hasDriver && Telemetry:GrpcListenPort > 0` | Only on a **fused** `admin,driver` node (e.g. docker-dev's central pair) — a driver-only node never registers the cookie stack at all, so it is structurally unaffected either way. |
|
||||
| `NodeCommandService.Execute` | same | Only on a fused node, same reasoning. |
|
||||
| `localdb_sync.v1.LocalDbSync/Sync` | `hasDriver && LocalDb:SyncListenPort > 0` | Only on a fused node that also sets a sync port — no node in the fleet currently has that shape, but the map is anonymised anyway. |
|
||||
|
||||
**A fused central pair is affected on its OWN dispatch, not only inbound from sites.** `CentralCommunicationActor` dials every enabled `ClusterNode` row, its own pair included, and central's telemetry dialer dials its own pair too — so a login-enabled fused central breaks its own command/telemetry plane, not just the sites'.
|
||||
|
||||
**Mapping mechanics.** All five maps live in one place, `MeshGrpcEndpoints.MapOtOpcUaMeshGrpcServices`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshGrpcEndpoints.cs`) — `Program.cs` calls it once and
|
||||
carries no map sites of its own. Four maps chain `.AllowAnonymous()` directly onto `MapGrpcService<T>()`.
|
||||
The fifth, `MapZbLocalDbSync()`, returns `IEndpointRouteBuilder` rather than an
|
||||
`IEndpointConventionBuilder`, so it cannot take the same chain; it is wrapped in an empty-prefix
|
||||
`app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync()`, which carries the convention without
|
||||
changing the wire route pattern.
|
||||
|
||||
**Three standing guards**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/`, so a future map
|
||||
cannot silently reopen this:
|
||||
|
||||
- `MeshGrpcAnonymousMetadataTests` — an endpoint-metadata census that calls the SAME
|
||||
`MapOtOpcUaMeshGrpcServices` production helper across all three role shapes (admin-only, driver-only,
|
||||
fused) and asserts every mesh endpoint carries `IAllowAnonymous`, with an exact expected count per shape
|
||||
and a scope check that the opt-out never leaks onto an unrelated endpoint.
|
||||
- `MeshGrpcMapConventionTests` — a `Program.cs` source scan asserting zero direct `MapGrpcService</
|
||||
MapZbLocalDbSync(` occurrences and exactly one call to the mapper, so a map added straight into
|
||||
`Program.cs` (bypassing the helper, and therefore invisible to the census) fails `MeshGrpcMapConventionTests`.
|
||||
- `MeshGrpcAuthStackTests` / `MeshGrpcDriverOnlyPipelineTests` drive the mesh endpoints through the REAL
|
||||
`AddOtOpcUaAuth` + `UseAuthentication`/`UseAuthorization` pipeline (`DisableLogin=false`) and the
|
||||
driver-only (no-auth-at-all) shape respectively — the CI-shaped reproduction of the outage, with a
|
||||
correctly-keyed call **succeeding** as the load-bearing assertion (a blocked call and a refused call both
|
||||
"fail"; only a genuine success through the real cookie stack proves the opt-out works) and an
|
||||
un-anonymised negative control proving the fixture's auth stack is actually live.
|
||||
|
||||
**Rig:** `docker-dev/docker-compose.yml`'s central-1/central-2 `Security__Auth__DisableLogin` is
|
||||
`${OTOPCUA_DISABLE_LOGIN:-true}` — default unchanged, but `OTOPCUA_DISABLE_LOGIN=false docker compose
|
||||
... up -d central-1 central-2` drives the rig through the real cookie login (`alice`/`bob`, password
|
||||
`password`, against the shared GLAuth) for exactly this kind of verification, instead of every prior mesh
|
||||
live gate running with the one middleware that breaks production neutralized.
|
||||
|
||||
See `archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md` (R3-01) for the full verification and design
|
||||
record, and `docs/Configuration.md` §`MeshTransport` / §`ConfigSource`-`ConfigServe` and `docs/Telemetry.md`
|
||||
for the affected listener ports.
|
||||
|
||||
---
|
||||
|
||||
## OTOPCUA0001 Analyzer — Compile-Time Guard
|
||||
|
||||
Per-capability resilience (retry, timeout, circuit-breaker) is applied by `CapabilityInvoker` in `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/`. A driver-capability call made **outside** the invoker bypasses resilience entirely — which in production looks like inconsistent timeouts, un-wrapped retries, and unbounded blocking.
|
||||
|
||||
Reference in New Issue
Block a user