scadaproj/otopcua-uns-loader/README.md

# otopcua-uns-loader

A **reloadable** populate-and-verify tool for the OtOpcUa galaxy Unified Namespace.
Recreates a UNS load grounded in the real AVEVA Galaxy **DEV** hierarchy (the 40
`TestMachine` instances) and verifies it streams **live values** on OPC UA — so
you can rebuild the OtOpcUa docker-dev instance and get the namespace back with
one populate + one deploy click.

## What it loads

One **SystemPlatform** `Tag` per `(machine, signal)` bound to the existing
`GalaxyMxGateway` driver (`MAIN-galaxy-mxgw`). Each tag's `FolderPath` is the
Galaxy object and its `Name` the attribute, so the materialised OPC UA variable
`OtOpcUa/<machine>/<signal>` has a NodeId equal to the MXAccess reference the
driver subscribes to — which is what makes the value go live.

Signals mirrored per machine (only those the instance actually has): the
`$TestMachine` process UDAs — `TestChangingInt`, `TestHistoryValue`,
`TestDouble/Float/Duration/DateTime`, `ProtectedValue(1)`, `TestAlarm001..003`,
`InAlarm` → **396 tags across 40 machines**.

Every row carries the `nw-mirror-` `TagId` prefix, so `clean` removes exactly
what the tool created (adopting any pre-existing seed row for the same ref).

## Why a deploy click is in the middle

OtOpcUa applies config only from **sealed Deployment snapshots**, and the only
way to seal one is the AdminUI **"Deploy current configuration"** button
(`http://localhost:9200/deployments`) — there is no SQL/REST/CLI trigger (it's
an in-cluster Akka operation). So the flow is:

```
populate ──SQL──▶ live config tables
                    │  (you click Deploy at :9200, sign in multi-role/password)
                    ▼
   driver applies ▶ materialises variables ▶ SubscribeBulk ▶ live values
                    │
verify ──OPC UA──▶ browse + read Good values on :4840
```

`populate` and `clean` print the reminder; `verify --wait` polls until the
deploy lands.

> Live values depend on the driver **SubscribeBulk** pass
> (OtOpcUa `master` ≥ commit `c1ce583`). On older builds variables materialise
> but stay `BadWaitingForInitialData`.

## Setup

```bash
cd otopcua-uns-loader
python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt
```

## Use

```bash
./.venv/bin/python otopcua_uns.py generate   # build load-plan.json from galaxy-hierarchy.json
./.venv/bin/python otopcua_uns.py populate    # upsert the 396 mirror Tag rows (idempotent)
#   → open http://localhost:9200/deployments, sign in, click "Deploy current configuration"
./.venv/bin/python otopcua_uns.py verify --wait   # poll until live values are Good on :4840
./.venv/bin/python otopcua_uns.py status      # config-DB + address-space snapshot
./.venv/bin/python otopcua_uns.py clean       # remove all nw-mirror-* tags (then Deploy again)
```

### Rebuild recovery (the point of the tool)

After the docker-dev instance is rebuilt (DB wiped):

1. Ensure the schema + clusters + the `MAIN-galaxy-mxgw` driver exist
   (`dotnet ef database update` + the docker-dev `cluster-seed`; see the OtOpcUa
   `docker-dev/README.md`).
2. `populate` → **Deploy** at the AdminUI → `verify --wait`.

## Troubleshooting

**`verify` stays INCOMPLETE / deployment "Sealed" but drivers never applied it.**
If you recreate the **admin/coordinator** node (`admin-a`) around the same time
you click Deploy, the dispatch broadcast can be lost — the deployment seals but
`NodeDeploymentState` shows no row for the driver nodes, and the address space
keeps the old content. Recover by: restart the driver nodes
(`docker restart otopcua-dev-driver-a-1 otopcua-dev-driver-b-1`) so they cleanly
re-subscribe, then Deploy again. If a no-op "NoChanges" blocks the re-deploy,
delete the orphan sealed `Deployment` row that no node applied (it has no
`NodeDeploymentState` children) so Deploy sees drift again.

**A rebuilt/restarted node serves an empty address space until the next Deploy.**
On bootstrap a node recovers to its last-*applied* revision and does **not**
re-materialise until a new deployment is dispatched — so after any node restart,
click Deploy once (a config change bumps the revision) to repopulate + re-subscribe.

## Configuration

Defaults target docker-dev; override via flags or env:

| Flag | Env | Default |
|---|---|---|
| `--sql-host/-port/-user/-password/-db` | `OTOPCUA_SQL_*` | `localhost:14330` sa / `OtOpcUa!Dev123` / `OtOpcUa` |
| `--opcua-endpoint` | `OTOPCUA_OPCUA_ENDPOINT` | `opc.tcp://localhost:4840` |
| `--driver` | `OTOPCUA_GALAXY_DRIVER` | `MAIN-galaxy-mxgw` |
| `--galaxy-json` | `OTOPCUA_GALAXY_JSON` | `../galaxy-hierarchy.json` |
| `--deploy-url` | — | `http://localhost:9200/deployments` |

## Files

- `otopcua_uns.py` — the CLI (generate / populate / verify / status / clean)
- `load-plan.json` — generated load plan (machine → signal → MXAccess ref)
- `../galaxy-hierarchy.json` — the source of truth, pulled live from the gateway
- `requirements.txt`, `.venv/`

## Company-shape overlay (`populate-equipment`)

Besides the galaxy-native mirror, the tool can load the **Northwind company
shape** (`filling / line-1 / rinser-01 / speed-rpm`) as a second, **Equipment**-kind
namespace (`nw-uns`, in cluster `MAIN`) from `../company-uns.json`. Each company
signal is a **VirtualTag** (+ a `Script`) whose script simply mirrors the live
galaxy-mirror tag for that signal:

```csharp
return ctx.GetTag("TestMachine_001.TestDouble").Value;
```

so the company shape carries live **VALUES** driven off the same Galaxy source — no
driver, no `BadWaitingForInitialData` once the galaxy mirror is up. The `ctx.GetTag`
literal is the signal's `source.fullTagReference`; the engine's `DependencyExtractor`
harvests it and subscribes the VirtualTag to that galaxy-mirror tag. This needs
OtOpcUa `master` ≥ the Equipment-namespace VirtualTag materialisation milestone (WS-3),
which materialises `VirtualTag`/`Script` rows on deploy and added the **headless
deploy** endpoint.

```bash
./.venv/bin/python otopcua_uns.py populate-equipment        # 3 areas / 8 lines / 40 equipment / 1036 VirtualTags
curl -s -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key'   # headless deploy
./.venv/bin/python otopcua_uns.py verify-equipment --expect 1036 --require-good 396 --wait --wait-seconds 300   # structure + live values
```

> **Verified live 2026-06-07** (OtOpcUa `feat/equipment-namespace-live-values`): galaxy mirror
> **396/396 Good**, company overlay **396 Good** on `opc.tcp://localhost:4840`, `VERIFY-EQUIPMENT: PASS`.
> Why 396 of 1036? The shipped `company-uns.json` invents **1036 distinct** `ctx.GetTag` refs, but only
> **396** of them match a real galaxy-mirror tag — so 396 signals are backed by a live source (and all 396
> go Good); the other 640 cite synthetic refs with no galaxy tag (`BadNodeIdUnknown`). That ratio is a
> property of the company model, not the streaming path — **every signal with a resolvable live source
> streams Good.** So `--require-good 396` is the meaningful gate for the current model. Survives a node
> restart with no re-deploy (the bootstrap-restore path re-materialises + re-applies the VirtualTags).

UNS folders carry the friendly **DisplayName** (`filling`); the BrowseName/NodeId
stay the stable logical Id (`nw-area-filling`) — standard OPC UA. **No driver:** the
company signals are VirtualTags (which link to Equipment + a Script, not a driver); a
placeholder `nw-uns-modbus` driver is kept only because an Equipment namespace is
expected to have one, but no `Tag` binds to it. `verify-equipment --require-good N`
reads each leaf's value and asserts at least N are Good (default `0` = structure-only,
back-compat); `--wait` polls until the deploy + change-triggered evaluations land.
Tracked in `OtOpcUa/docs/plans/2026-06-06-equipment-namespace-materialization-scope.md` (WS-3).

`clean` removes both the mirror tags and the company overlay (the `VirtualTag` +
`Script` rows, in FK-safe order, plus the namespace/driver/equipment/areas/lines).