Add Galaxy repository API and clients
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,271 @@
|
||||
# Galaxy Repository Browse
|
||||
|
||||
The gateway exposes a read-only browse surface over the AVEVA System Platform
|
||||
Galaxy Repository (the SQL Server database named `ZB`). Clients use it to
|
||||
enumerate the deployed object hierarchy and each object's dynamic attributes
|
||||
before subscribing to runtime values via the existing `MxAccessGateway` RPCs.
|
||||
|
||||
This is a metadata layer: it never reads or writes runtime tag values, never
|
||||
goes through MXAccess COM, and never needs the x86 worker. It runs entirely
|
||||
inside the .NET 10 gateway process and talks to the Galaxy Repository over
|
||||
plain SQL.
|
||||
|
||||
## Why It Exists
|
||||
|
||||
Without browse, a client must already know `tag_name.AttributeName` strings to
|
||||
issue `AddItem`. The Galaxy Repository is the authoritative source for those
|
||||
names — the same database that System Platform itself reads when the
|
||||
ArchestrA IDE renders the deployment tree. Surfacing that data over gRPC lets
|
||||
remote clients build a navigable address space without any coupling to the
|
||||
COM layer or the host platform.
|
||||
|
||||
The query bodies are kept byte-for-byte identical to the equivalent OPC UA
|
||||
server in the OtOpcUa project so the two consumers see the same row sets.
|
||||
|
||||
## RPC Surface
|
||||
|
||||
The service is defined in
|
||||
`src/MxGateway.Contracts/Protos/galaxy_repository.proto` under package
|
||||
`galaxy_repository.v1`.
|
||||
|
||||
| RPC | Purpose |
|
||||
|-----|---------|
|
||||
| `TestConnection` | Connectivity probe. Returns `{ ok: bool }` after a `SELECT 1`. Does not throw on SQL failure — returns `ok = false`. Always hits SQL directly so it remains a true health check. |
|
||||
| `GetLastDeployTime` | Returns the cached `galaxy.time_of_last_deploy`. Served from the shared hierarchy cache; refreshed in the background. |
|
||||
| `DiscoverHierarchy` | Returns the full deployed hierarchy plus every object's dynamic attributes. **Served from cache** — see [Hierarchy Cache](#hierarchy-cache). |
|
||||
| `WatchDeployEvents` | **Server-streaming.** The server emits the current state immediately on subscribe (so clients can bootstrap without waiting), then emits one event per detected deploy change. See [Deploy Notifications](#deploy-notifications). |
|
||||
|
||||
`DiscoverHierarchy` is intentionally a single unary RPC rather than a stream:
|
||||
the row set is small (thousands of objects, low tens-of-thousands of
|
||||
attributes for typical Galaxies) and clients almost always want the whole tree
|
||||
at once.
|
||||
|
||||
## Hierarchy Cache
|
||||
|
||||
The gateway holds a single shared `IGalaxyHierarchyCache`
|
||||
(`src/MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
|
||||
`DiscoverHierarchy` and `GetLastDeployTime` request reads from this cache
|
||||
rather than hitting SQL. Many clients can browse concurrently with at most
|
||||
one SQL query in flight.
|
||||
|
||||
Refresh strategy is **deploy-time gated**:
|
||||
|
||||
1. The hosted `GalaxyHierarchyRefreshService` ticks every
|
||||
`MxGateway:Galaxy:DashboardRefreshIntervalSeconds` seconds (default 30).
|
||||
2. Each tick queries the cheap `SELECT time_of_last_deploy FROM galaxy` first.
|
||||
3. If the deploy timestamp is unchanged, the heavy hierarchy + attributes
|
||||
queries are **skipped**. The cache simply marks `LastSuccessAt`.
|
||||
4. If the deploy timestamp changed (or no data has loaded yet), the cache
|
||||
pulls hierarchy + attributes, materializes a `DiscoverHierarchyReply`
|
||||
once, replaces the entry atomically, and publishes a deploy event.
|
||||
|
||||
Materializing the reply at refresh time means subsequent `DiscoverHierarchy`
|
||||
calls return a pre-built proto message — no per-request projection, no
|
||||
per-request allocations beyond the gRPC serializer's frame.
|
||||
|
||||
When SQL is unreachable, the cache retains the previous data and flips
|
||||
`Status` to `Stale` (or `Unavailable` if no data was ever loaded). A
|
||||
`SqlException` never bubbles out as the client-facing error.
|
||||
|
||||
### First-load behavior
|
||||
|
||||
If a client calls `DiscoverHierarchy` before the background service has
|
||||
populated the cache, the gRPC handler waits up to 5 seconds for the first
|
||||
load to complete before returning. If the first load fails or times out,
|
||||
the client gets `Unavailable` with a short reason. Once any load completes
|
||||
(success or failure), this wait is skipped on subsequent calls.
|
||||
|
||||
## Deploy Notifications
|
||||
|
||||
`WatchDeployEvents` is a server-streaming RPC backed by
|
||||
`IGalaxyDeployNotifier` (`src/MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`).
|
||||
The notifier maintains a private bounded channel per subscriber so a slow
|
||||
client cannot back-pressure other subscribers or the publisher.
|
||||
|
||||
Subscriber lifecycle:
|
||||
|
||||
1. On subscribe, the notifier emits the **current** event (the state of the
|
||||
most recent successful refresh) so the subscriber can sync its local cache
|
||||
without waiting for the next deploy. Clients that already know about that
|
||||
deploy can pass `last_seen_deploy_time` in the request to suppress the
|
||||
bootstrap event.
|
||||
2. As the cache observes new deploy timestamps, it publishes one event per
|
||||
change. Each event carries:
|
||||
- `sequence` — monotonic per server start; gaps signal a dropped event.
|
||||
- `observed_at` — server wall-clock when the cache saw the deploy.
|
||||
- `time_of_last_deploy` (+ `time_of_last_deploy_present`) — the Galaxy
|
||||
timestamp; absent only when the source row reports null.
|
||||
- `object_count`, `attribute_count` — counts on the new deploy, useful for
|
||||
dashboards and "did anything important change" gates without re-pulling.
|
||||
3. If the subscriber's per-subscriber buffer fills (bound = 16 events with
|
||||
`DropOldest`), older events are dropped. Clients use the `sequence` field
|
||||
to detect this.
|
||||
4. Cancellation (or transport disconnect) removes the subscriber.
|
||||
|
||||
Typical client pattern:
|
||||
|
||||
```text
|
||||
1. Open WatchDeployEvents stream (with last_seen_deploy_time if you have one).
|
||||
2. On each event, decide whether to call DiscoverHierarchy to refresh local cache.
|
||||
3. If sequence skipped a number, treat it as a dropped event and refresh.
|
||||
```
|
||||
|
||||
### Reply Shape
|
||||
|
||||
```proto
|
||||
message GalaxyObject {
|
||||
int32 gobject_id = 1;
|
||||
string tag_name = 2;
|
||||
string contained_name = 3;
|
||||
string browse_name = 4; // contained_name when present, else tag_name
|
||||
int32 parent_gobject_id = 5;
|
||||
bool is_area = 6;
|
||||
int32 category_id = 7;
|
||||
int32 hosted_by_gobject_id = 8;
|
||||
repeated string template_chain = 9;
|
||||
repeated GalaxyAttribute attributes = 10;
|
||||
}
|
||||
|
||||
message GalaxyAttribute {
|
||||
string attribute_name = 1;
|
||||
string full_tag_reference = 2; // e.g. "DelmiaReceiver_001.DownloadPath"
|
||||
int32 mx_data_type = 3; // raw Galaxy mx_data_type integer
|
||||
string data_type_name = 4;
|
||||
bool is_array = 5;
|
||||
int32 array_dimension = 6;
|
||||
bool array_dimension_present = 7; // distinguishes "no dimension" from 0
|
||||
int32 mx_attribute_category = 8;
|
||||
int32 security_classification = 9;
|
||||
bool is_historized = 10;
|
||||
bool is_alarm = 11;
|
||||
}
|
||||
```
|
||||
|
||||
### Contained Name vs Tag Name
|
||||
|
||||
Galaxy objects carry two names. `tag_name` is globally unique and is what
|
||||
MXAccess expects in `AddItem`. `contained_name` is the human-readable name
|
||||
used in the IDE browse tree, scoped to the parent. The browse RPC exposes
|
||||
both: clients display `browse_name` to users and pass `tag_name` (or
|
||||
`full_tag_reference`) into MXAccess subscriptions. When `contained_name` is
|
||||
empty (top-level objects), `browse_name` falls back to `tag_name`.
|
||||
|
||||
### Data Types
|
||||
|
||||
`mx_data_type` is returned as the raw Galaxy integer rather than mapped to a
|
||||
language-neutral enum. The gateway makes no assumption about the client's
|
||||
target type system — clients map to OPC UA, JSON, .NET CLR types, or
|
||||
something else as appropriate. The Galaxy `data_type` table description is
|
||||
also passed through as `data_type_name`.
|
||||
|
||||
`array_dimension_present` is a separate boolean because protobuf scalar
|
||||
fields cannot express null. Use it to distinguish "no dimension reported" from
|
||||
"dimension is zero."
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
gRPC client(s)
|
||||
-> GalaxyRepositoryGrpcService (src/MxGateway.Server/Grpc/)
|
||||
DiscoverHierarchy, GetLastDeployTime -> IGalaxyHierarchyCache.Current
|
||||
WatchDeployEvents -> IGalaxyDeployNotifier
|
||||
TestConnection -> GalaxyRepository (direct SQL)
|
||||
|
||||
GalaxyHierarchyRefreshService (BackgroundService)
|
||||
-> IGalaxyHierarchyCache.RefreshAsync
|
||||
-> GalaxyRepository.GetLastDeployTimeAsync (cheap, every tick)
|
||||
-> GalaxyRepository.GetHierarchyAsync (only on deploy change)
|
||||
-> GalaxyRepository.GetAttributesAsync (only on deploy change)
|
||||
-> GalaxyProtoMapper.MapObject (materialize DiscoverHierarchyReply once)
|
||||
-> IGalaxyDeployNotifier.Publish (only on deploy change)
|
||||
```
|
||||
|
||||
Component breakdown:
|
||||
|
||||
- `GalaxyRepository` (`src/MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
|
||||
the SQL. Its constants `HierarchySql` and `AttributesSql` are copied verbatim
|
||||
from the OtOpcUa project; do not edit them in isolation here. The two
|
||||
queries walk template-derivation and package-derivation chains via
|
||||
recursive CTEs and pick the most-derived attribute override per object.
|
||||
- `GalaxyHierarchyCache`
|
||||
(`src/MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
|
||||
recent immutable `GalaxyHierarchyCacheEntry` (rows + materialized proto
|
||||
reply + counts + status). All gRPC clients share the same entry.
|
||||
- `GalaxyHierarchyRefreshService`
|
||||
(`src/MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
|
||||
hosted `BackgroundService` that drives `RefreshAsync` on the configured
|
||||
interval, with deploy-time gating to avoid unnecessary heavy queries.
|
||||
- `GalaxyDeployNotifier`
|
||||
(`src/MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
|
||||
per-subscriber-channel fan-out for streaming clients.
|
||||
- `GalaxyProtoMapper`
|
||||
(`src/MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
|
||||
proto messages. Used by the cache during refresh to materialize the reply
|
||||
once.
|
||||
- `GalaxyRepositoryGrpcService`
|
||||
(`src/MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`) implements
|
||||
the four RPCs.
|
||||
|
||||
## Configuration
|
||||
|
||||
Bound to `MxGateway:Galaxy` via `GalaxyRepositoryOptions`.
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Galaxy:ConnectionString` | `Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;` | SQL Server connection string for the Galaxy Repository. Integrated Security against `localhost` is the dev default; production deployments should override this through the standard double-underscore environment variable form, e.g. `MxGateway__Galaxy__ConnectionString`. |
|
||||
| `MxGateway:Galaxy:CommandTimeoutSeconds` | `60` | Per-command SQL timeout. Applies to all three RPCs. |
|
||||
|
||||
The connection string is not treated as a secret in dev (`Integrated
|
||||
Security`), but production deployments that use SQL authentication should set
|
||||
the override via environment variable rather than committing credentials to
|
||||
`appsettings.json`.
|
||||
|
||||
## Authorization
|
||||
|
||||
All four Galaxy RPCs (including `WatchDeployEvents`) require the
|
||||
`metadata:read` API-key scope. Browse is read-only metadata, equivalent in
|
||||
privilege to `MxCommandKind.GetSessionState` or `MxCommandKind.GetWorkerInfo`.
|
||||
The mapping lives in `GatewayGrpcScopeResolver`; see
|
||||
[Authorization](./Authorization.md) for the full scope catalog.
|
||||
|
||||
A request without an API key returns `Unauthenticated`. A request with a key
|
||||
that lacks `metadata:read` returns `PermissionDenied` with the missing scope
|
||||
embedded in the status detail.
|
||||
|
||||
## Dashboard Surface
|
||||
|
||||
The gateway's Blazor dashboard surfaces a Galaxy summary in two places:
|
||||
|
||||
- An overview card on `/dashboard` showing connectivity status, last deploy
|
||||
timestamp, object count (with area count), attribute total, historized and
|
||||
alarm counts, and last successful refresh.
|
||||
- A dedicated `/dashboard/galaxy` page with object-category and top-template
|
||||
breakdowns plus a Sync Info table covering last successful refresh, last
|
||||
attempt, refresh interval, redacted connection string, and command timeout.
|
||||
|
||||
Both views are projected from the same `IGalaxyHierarchyCache` that backs the
|
||||
gRPC service. The dashboard does not run its own refresh — when the
|
||||
background `GalaxyHierarchyRefreshService` updates the cache, both the
|
||||
overview card and the `/dashboard/galaxy` page pick up the new state on the
|
||||
next dashboard tick. When SQL is unreachable, the cache retains the previous
|
||||
data and flips `Status` to `Stale` or `Unavailable`; the dashboard surfaces
|
||||
that as a yellow or red status badge plus the truncated error.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- The service is registered alongside `MxAccessGatewayService` in
|
||||
`GatewayApplication.MapGatewayEndpoints`. Both services share the same
|
||||
authorization interceptor and authentication policy.
|
||||
- Failures to reach the Galaxy database surface as `Unavailable`. Detailed
|
||||
SQL exceptions are logged at `Warning` and never returned to clients.
|
||||
- Integration tests live in
|
||||
`src/MxGateway.IntegrationTests/Galaxy/GalaxyRepositoryLiveTests.cs`. Set
|
||||
`MXGATEWAY_RUN_LIVE_GALAXY_TESTS=1` (and optionally
|
||||
`MXGATEWAY_LIVE_GALAXY_CONN`) to run them; otherwise they skip.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Contracts](./Contracts.md)
|
||||
- [Grpc](./Grpc.md)
|
||||
- [Authorization](./Authorization.md)
|
||||
- [Gateway Configuration](./GatewayConfiguration.md)
|
||||
Reference in New Issue
Block a user