Files
mxaccessgw/clients/python/README.md
T
Joseph Doherty a8f86b5336
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m19s
ci / portable (push) Successful in 23m59s
ci / windows-x86 (push) Failing after 58m42s
test(tst-24): drive the .NET and Python clients against real in-process gRPC servers
TST-24 asked for per-client wire tests against a fake gateway. An audit first
corrected the finding's premise: Go, Rust, and Java already had them — bufconn,
a loopback tonic server, and InProcessServerBuilder respectively — each already
asserting the round trip, the server-observed bearer header, and the ReplayGap
sentinel. The two genuine gaps were .NET (every test substituted the transport
interface; the test project had no server package at all) and Python (stub
monkeypatching everywhere but one opt-in TLS test).

Both now serve mxaccess_gateway.v1.MxAccessGateway over a real transport —
Kestrel h2c and grpc.aio, each on an ephemeral loopback port — and drive the
ordinary public client API against it. Only the gateway's behaviour is canned;
the framing, serialization, metadata, and status codes are genuine. Four shapes
each: full round trip with every reply field asserted, the authorization header
as received by the server (including on the streaming RPC), the ReplayGap
sentinel surfaced as the client's typed signal, and a real PERMISSION_DENIED
mapping to the typed authorization error.

The .NET client was only ever compiled in CI, never tested, so the portable job
gains a dotnet test step.

Fixes a bug the new tests caught on their first run: Python's connect() built the
grpc.aio channel inside asyncio.to_thread, and a grpc.aio channel binds to the
event loop current on the constructing thread, so every non-stub connection
raised 'There is no current event loop in thread'. No mock-based test could see
it, and the test guarding the off-loop behaviour patched create_channel and so
asserted the bug. Split resolve_channel_security (blocking TOFU probe, off-loop)
from create_channel (on-loop); the guard tests now assert both halves.
2026-08-10 08:22:25 -04:00

486 lines
19 KiB
Markdown

# Python Client
The Python client package contains generated MXAccess Gateway protobuf
bindings, the async `zb_mom_ww_mxgateway` package, and the `mxgw-py` test CLI. The
package uses the shared proto inputs documented in
`../../docs/ClientProtoGeneration.md` so gateway and client contracts stay in
sync.
## Layout
```text
clients/python/
pyproject.toml
generate-proto.ps1
src/zb_mom_ww_mxgateway/
src/zb_mom_ww_mxgateway/generated/
src/zb_mom_ww_mxgateway_cli/
tests/
```
`src/zb_mom_ww_mxgateway/generated` contains code produced by `grpc_tools.protoc`. Do not
edit generated files by hand.
## Regenerating Protobuf Bindings
Run generation after the shared `.proto` files or the Python output path
changes:
```powershell
./generate-proto.ps1
```
The script uses the Python tool path recorded in
`../../docs/ToolchainLinks.md`.
## Build And Test
Run the Python checks from `clients/python`:
```powershell
python -m pip install -e ".[dev]"
python -m pytest
python -m pip wheel . --no-deps --wheel-dir "$env:TEMP\mxgateway-python-wheel"
```
The tests import the generated gateway and worker stubs, run fake async gateway
stubs, verify API key metadata, exercise stream cancellation, load shared value
and command fixtures, and check deterministic CLI output.
`tests/test_wire_fake_gateway.py` is the one suite that does **not** substitute a
stub: it serves a canned `MxAccessGatewayServicer` from a real `grpc.aio` server
on an ephemeral loopback port and drives the ordinary `GatewayClient` API against
it. Only the gateway's behaviour is canned — the HTTP/2 framing, protobuf
serialization, `authorization` metadata, and gRPC status codes are genuine, so it
catches decode and metadata breaks a stub fake cannot see. No MXAccess, no worker,
no TLS, so it runs in the default suite. See `docs/GatewayTesting.md`
(Client Wire Tests) for the cross-client pattern.
```powershell
python -m pytest tests/test_wire_fake_gateway.py
```
## Packaging
Install the package in editable mode for local development:
```powershell
python -m pip install -e ".[dev]"
```
Build a wheel from `clients/python`:
```powershell
python -m pip wheel . --no-deps --wheel-dir "$env:TEMP\mxgateway-python-wheel"
```
Install the generated wheel into a target environment:
```powershell
python -m pip install <wheel-path>
```
The wheel exposes the `mxgw-py` console script.
## Library Usage
The library is async-first:
```python
from zb_mom_ww_mxgateway import GatewayClient
async with await GatewayClient.connect(
endpoint="localhost:5000",
api_key="<gateway-api-key>",
plaintext=True,
) as client:
session = await client.open_session(client_session_name="python-client")
try:
server_handle = await session.register("python-client")
item_handle = await session.add_item(server_handle, "Object.Attribute")
await session.advise(server_handle, item_handle)
finally:
await session.close()
```
`GatewayClient.open_session_raw`, `GatewayClient.invoke_raw`, and
`GatewayClient.stream_events_raw` keep the generated protobuf replies and
events available for parity tests. `Session` helpers call the method-specific
MXAccess commands and preserve raw replies on typed command exceptions.
For alarms, the client exposes `GatewayClient.query_active_alarms` (one-shot
snapshot), `GatewayClient.stream_alarms` (async generator yielding alarm-feed
messages from the gateway's central monitor), and
`GatewayClient.acknowledge_alarm` (ack by alarm reference, optional comment
and ack target). Cancel the surrounding task or `aclose()` the iterator to
terminate the stream.
Canceling a Python task cancels the client-side gRPC call or stream wait. It
does not abort an in-flight MXAccess COM call inside the worker process.
### Event streaming and reconnect gaps
`Session.stream_events()` yields an async iterator whose items are either a
normal `MxEvent` or a `ReplayGap`. Track the `worker_sequence` of the last event
you processed and pass it back as `after_worker_sequence` to resume after a
disconnect:
```python
from zb_mom_ww_mxgateway import ReplayGap
cursor = 0
async for item in session.stream_events(after_worker_sequence=cursor):
if isinstance(item, ReplayGap):
# The gateway dropped events between item.requested_after_sequence and
# item.oldest_available_sequence — they are gone from the replay ring.
# Discard local tag/alarm state and re-snapshot (e.g. read_bulk /
# query_active_alarms), then resume without another gap:
cursor = item.resume_after_worker_sequence # oldest_available_sequence - 1
continue
# Normal MXAccess event.
cursor = item.worker_sequence
handle(item)
```
`ReplayGap` is the gateway's reconnect-replay gap sentinel made typed and
observable. It is delivered only at the head of a stream resumed with a non-zero
`after_worker_sequence` when the requested cursor predates the oldest retained
event. It is a **non-terminal** signal — the stream continues with normal events
after it — and it is never yielded as an `MxEvent`, so it can never be mistaken
for a real MXAccess event. The client does not synthesize or swallow it; the
gateway only sets it on `StreamEvents` results (never on a fresh stream or a
`DrainEvents` reply). `GatewayClient.stream_events_raw` remains the raw protobuf
stream (the sentinel arrives there as an `MxEvent` with `replay_gap` set).
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
forwards them unchanged — it does not paper over them.
### Attributing a write to a user without `authenticate_user`
MXAccess only stamps a plain `write`/`write2` with a Galaxy user id when the
item carries an active *supervisory* advise. If you are **not** using the
verified/secured path (`authenticate_user``write_secured`/`write_secured2`)
but still need the write attributed to a user id, you must first advise the
item supervisory and then pass that user id on the write. Without the
supervisory advise the `user_id` on a plain write is ignored.
The session exposes a typed `advise_supervisory` helper alongside
`advise`/`unadvise`:
```python
await session.advise_supervisory(server_handle, item_handle)
await session.write(server_handle, item_handle, value, user_id=user_id)
```
The CLI exposes the same command as `advise-supervisory`, and `write` /
`write2` take `--user-id`.
For the verified/secured path, `authenticate_user`, `write_secured`,
`write_secured2`, and `archestra_user_to_id` are typed session helpers too. The
credential passed to `authenticate_user` and the values written by
`write_secured`/`write_secured2` are treated as secrets: they are never logged
and are scrubbed from any surfaced error message. MXAccess parity is preserved —
a `write_secured` that fails because no prior `authenticate_user` +
`advise_supervisory` established a supervisory context surfaces the native
failure as `MxAccessError` rather than being silently "fixed":
```python
user_id = await session.authenticate_user(server_handle, "operator", password)
await session.advise_supervisory(server_handle, item_handle)
await session.write_secured(
server_handle,
item_handle,
value,
current_user_id=user_id,
verifier_user_id=user_id,
)
```
The CLI mirrors these as `authenticate-user` (credential via `--password` or,
preferably, the variable named by `--password-env`, default
`MXGATEWAY_VERIFY_PASSWORD`) and `write-secured`. The credential is required: a
missing or empty resolved value raises a `UsageError` naming the option and the
variable, so the CLI fails before connecting instead of authenticating with an
empty password. `MXGATEWAY_VERIFY_PASSWORD` is the canonical variable across all
five client CLIs — see
[Cross-Language Smoke Matrix](../../docs/CrossLanguageSmokeMatrix.md).
### Array writes replace the whole array
A write to an array attribute **replaces the entire array**; it is not an
element-wise patch. To change a subset of elements, send the full array with
the unchanged elements included. For example, to change 2 elements of a
20-element array, build the `MxValue` from all 20 values (the 18 unchanged plus
the 2 new ones). Sending only the 2 changed values overwrites the attribute
with a 2-element array.
### Default-fill partial array writes
`Session.write_array_elements` lets you write only the indices you care about.
The gateway fills every unmentioned position with the type default for the
declared `element_data_type` (0, `False`, `""`, Unix epoch for timestamps).
The previous value at those positions is **not** preserved — the gateway expands
the sparse map to a full array before forwarding the write to MXAccess, so this
is still a full replacement:
```python
# Write indices 0 and 5 of a 10-element integer array.
# Positions 1-4 and 6-9 become 0, not their previous values.
await session.write_array_elements(
server_handle=server_handle,
item_handle=item_handle,
element_data_type=pb.MX_DATA_TYPE_INTEGER,
total_length=10,
elements={0: 100, 5: 500},
)
```
Bare-name array items (e.g. `Object.ArrayAttr` without an index suffix) added
via `add_item`, `add_item2`, `add_item_bulk`, or `add_buffered_item`
auto-normalize to `[]` — they refer to the whole array, not a single element.
Writes through such handles must cover the full array or use
`write_array_elements` to supply `total_length` and let the gateway fill
defaults for the rest.
## Galaxy Repository Browse
The `GalaxyRepositoryClient` wraps the read-only `GalaxyRepository` gRPC
service. It lets callers test connectivity to the AVEVA System Platform
Galaxy Repository (ZB SQL database), read the last deploy timestamp, and
enumerate the deployed object hierarchy plus each object's dynamic
attributes:
```python
from zb_mom_ww_mxgateway import GalaxyRepositoryClient
async with await GalaxyRepositoryClient.connect(
endpoint="localhost:5000",
api_key="<gateway-api-key>",
plaintext=True,
) as galaxy:
if not await galaxy.test_connection():
raise RuntimeError("gateway cannot reach the Galaxy Repository DB")
last_deploy = await galaxy.get_last_deploy_time()
print(f"last deploy: {last_deploy}")
for obj in await galaxy.discover_hierarchy():
print(obj.tag_name, obj.contained_name)
for attr in obj.attributes:
print(" ", attr.attribute_name, "->", attr.full_tag_reference)
```
The methods return native Python types (`bool`, `datetime | None`, and a
`list[GalaxyObject]` of generated proto messages) so callers can index
into the hierarchy without learning the underlying stub class. The
service requires the `metadata:read` scope on the API key.
### Browsing lazily
For UI trees or OPC UA bridges, use `browse_children_raw` to walk one level at a
time instead of loading the full hierarchy with `discover_hierarchy`. Pass an
empty request for root objects; subsequent calls set `parent_gobject_id`,
`parent_tag_name`, or `parent_contained_path`. Filter fields match
`DiscoverHierarchy`. Each response pairs `children` with `child_has_children` so
you know which nodes to expand. Most callers should prefer the higher-level
`browse()` / `LazyBrowseNode` walker below; `browse_children_raw` is the
low-level escape hatch for direct page-token control. See
[Galaxy Repository](../../docs/GalaxyRepository.md#browsechildren) for full
request and filter semantics.
```python
from zb_mom_ww_mxgateway.generated import galaxy_repository_pb2 as galaxy_pb2
reply = await galaxy.browse_children_raw(galaxy_pb2.BrowseChildrenRequest())
for child, has_children in zip(reply.children, reply.child_has_children):
print(child.tag_name, "expand=" + str(has_children))
```
#### High-level walker
For UI trees, the client provides a `LazyBrowseNode` walker that handles
sibling pagination and the `child_has_children` hint for you:
```python
async with await GalaxyRepositoryClient.connect(
endpoint="localhost:5000",
api_key="<gateway-api-key>",
plaintext=True,
) as galaxy:
roots = await galaxy.browse()
for root in roots:
if root.has_children_hint:
await root.expand()
for child in root.children:
kind = "has children" if child.has_children_hint else "leaf"
print(f"{child.object.tag_name} ({kind})")
```
`expand` is idempotent — calling it twice fires only one RPC,
and is safe under concurrent callers. To refresh after a Galaxy redeploy, call
`browse` again from the root.
### Watching deploy events
`GalaxyRepositoryClient.watch_deploy_events` opens a server-streaming
subscription that emits the current cached deploy state immediately and
then one `DeployEvent` per new Galaxy deploy. `sequence` is monotonic per
gateway start; gaps mean events were dropped from the per-subscriber
buffer. Pass `last_seen_deploy_time` to suppress the bootstrap event when
the caller already has the current state cached:
```python
from datetime import datetime, timezone
from zb_mom_ww_mxgateway import DeployEvent, GalaxyRepositoryClient
async with await GalaxyRepositoryClient.connect(
endpoint="localhost:5000",
api_key="<gateway-api-key>",
plaintext=True,
) as galaxy:
last_seen: datetime | None = None
async for event in galaxy.watch_deploy_events(last_seen_deploy_time=last_seen):
assert isinstance(event, DeployEvent)
print(
f"#{event.sequence} deploy={event.time_of_last_deploy.ToDatetime(tzinfo=timezone.utc)} "
f"objects={event.object_count} attributes={event.attribute_count}"
)
if event.time_of_last_deploy_present:
last_seen = event.time_of_last_deploy.ToDatetime(tzinfo=timezone.utc)
```
The method returns an async iterator yielding the generated `DeployEvent`
proto. Breaking out of the loop, calling `aclose()` on the iterator, or
cancelling the surrounding task closes the underlying gRPC stream
cleanly. The streaming RPC requires the same `metadata:read` scope as
the other Galaxy methods.
The CLI exposes the Galaxy Repository RPCs through five subcommands that
mirror the other clients:
```bash
mxgw-py galaxy-test-connection --plaintext --json
mxgw-py galaxy-last-deploy --plaintext --json
mxgw-py galaxy-discover --plaintext --json
mxgw-py galaxy-browse --plaintext --json
mxgw-py galaxy-watch --plaintext --json
```
`galaxy-watch` is bounded by `--max-events` (default `1`) and `--timeout`
(seconds) so it always terminates; pass `--last-seen-deploy-time` (an
ISO-8601 timestamp) to suppress the bootstrap event when it matches the
current cached deploy time.
`galaxy-browse` wraps the lazy `LazyBrowseNode` walker. Without `--depth`
it lists only the root objects; `--depth N` eagerly expands `N` further
levels before printing. Text output is a node count followed by an indented
tree (`+`/`-` marks the server's has-children hint); `--json` emits nested
`{..., "hasChildrenHint": bool, "children": [...]}` nodes that match the
`galaxy-discover` object shape. The `BrowseChildrenRequest` filters are
exposed as `--category-id` (repeatable), `--template-chain-contains`
(repeatable), `--tag-name-glob`, `--include-attributes`,
`--alarm-bearing-only`, and `--historized-only`, all AND-combined.
## Authentication And TLS
`ClientOptions.api_key` adds this metadata to unary calls and streams:
```text
authorization: Bearer <api-key>
```
The client supports plaintext channels for local development, TLS with system
roots, TLS with a custom `ca_file`, and an optional test server name override.
API keys are redacted from option repr output and CLI error output.
The gateway can auto-generate its own self-signed certificate (it has no PKI).
grpc-python has no per-channel skip-verify, so the lenient TLS default is
**trust-on-first-use**: with no `ca_file` and `require_certificate_validation`
left `False`, the client fetches the gateway's presented certificate once
(unverified) and pins it for the channel, defaulting the SNI/target-name override
to `localhost` (the generated certificate always carries a `localhost` SAN) when
none was supplied. To verify instead, pass `ca_file` to verify against a specific
CA, or set `require_certificate_validation=True` to verify against the system
trust roots. The strict posture is reachable through every documented entry
point: the `require_certificate_validation=True` keyword on
`GatewayClient.connect(...)` / `GalaxyRepositoryClient.connect(...)`, the
`ClientOptions(require_certificate_validation=True)` struct, and the
`--require-certificate-validation` CLI flag. See
[Gateway Configuration](../../docs/GatewayConfiguration.md#automatic-self-signed-certificate).
Channel construction is split in two: `resolve_channel_security(options)` performs
the blocking part (the trust-on-first-use certificate probe) and
`create_channel(options, security=...)` builds the channel. The async `connect`
classmethods run the first off the event loop and the second on it, because a
`grpc.aio` channel binds to the event loop current on the constructing thread —
building it inside `asyncio.to_thread` raises
`RuntimeError: There is no current event loop in thread 'asyncio_N'`. Callers that
build their own channel should keep `create_channel` on the loop thread.
## CLI
The CLI emits deterministic JSON for automation:
```powershell
mxgw-py version --json
mxgw-py open-session --endpoint localhost:5000 --plaintext --json
mxgw-py register --session-id <id> --client-name python-client --json
mxgw-py add-item --session-id <id> --server-handle 1 --item Object.Attribute --json
mxgw-py advise --session-id <id> --server-handle 1 --item-handle 2 --json
mxgw-py stream-events --session-id <id> --max-events 1 --json
mxgw-py stream-alarms --max-messages 1 --json
mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --json
mxgw-py write --session-id <id> --server-handle 1 --item-handle 2 --type int32 --value 123 --json
```
Use `--api-key` or `--api-key-env MXGATEWAY_API_KEY` to attach API key
metadata. `smoke` opens a session, registers, adds an item, advises, streams a
bounded event count, and closes the session in a `finally` block.
Use TLS options for a secured gateway:
```powershell
mxgw-py smoke --endpoint mxgateway.example.local:5001 --tls --ca-file C:\certs\mxgateway-ca.pem --server-name-override mxgateway.example.local --api-key-env MXGATEWAY_API_KEY --item Object.Attribute --json
```
To force certificate validation against the system trust store instead of the
lenient trust-on-first-use default, add `--require-certificate-validation`:
```powershell
mxgw-py smoke --endpoint mxgateway.example.local:5001 --tls --require-certificate-validation --api-key-env MXGATEWAY_API_KEY --item Object.Attribute --json
```
## Integration Checks
Run live checks only when a gateway and MXAccess-backed worker are available:
```powershell
$env:MXGATEWAY_INTEGRATION = '1'
$env:MXGATEWAY_ENDPOINT = 'localhost:5000'
$env:MXGATEWAY_API_KEY = '<gateway-api-key>'
$env:MXGATEWAY_TEST_ITEM = 'Object.Attribute'
mxgw-py smoke --endpoint $env:MXGATEWAY_ENDPOINT --plaintext --api-key-env MXGATEWAY_API_KEY --item $env:MXGATEWAY_TEST_ITEM --json
```
## Installing from the Gitea PyPI Feed
The client publishes to the internal Gitea PyPI feed:
````bash
pip install \
--index-url https://gitea.dohertylan.com/api/packages/dohertj2/pypi/simple/ \
zb-mom-ww-mxaccess-gateway-client
````
If you need authentication (private feed), use `--extra-index-url` and either
a `~/.netrc` entry or `PIP_INDEX_URL=https://<user>:<token>@gitea.dohertylan.com/...`.
## Related Documentation
- [Client Packaging](../../docs/ClientPackaging.md)
- [Client Proto Generation](../../docs/ClientProtoGeneration.md)
- [Python Client Detailed Design](./PythonClientDesign.md)