Add Galaxy UNS artifacts + reloadable OtOpcUa loader tool

galaxy-hierarchy.json: full AVEVA Galaxy DEV hierarchy pulled live via the
MxGateway .NET client (129 objects, 14k attrs). company-uns.json/.tree.txt +
gen_uns.py: a fake-company (Northwind) ISA-95 UNS modeled on OtOpcUa's
Cluster->Namespace->Area->Line->Equipment->Tag schema, grounded in the 40
TestMachine instances. otopcua-uns-loader/: reloadable generate/populate/verify/
clean tool that recreates + verifies the galaxy mirror (396 live tags across 40
machines) in OtOpcUa's config DB after a rebuild.
This commit is contained in:
Joseph Doherty
2026-06-06 14:22:25 -04:00
parent e5a609be83
commit eb26bf3248
8 changed files with 134764 additions and 0 deletions
+19317
View File
File diff suppressed because it is too large Load Diff
+56
View File
@@ -0,0 +1,56 @@
Northwind Consumer Products — Unified Namespace
(generated from Galaxy DESKTOP-6JL3KKO\DEV; 40 machines, 1036 signals)
northwind
└─ birmingham
├─ filling/ (Filling & Capping; from Galaxy TestArea)
│ ├─ line-1/
│ │ ├─ rinser-01 [krones Hydra Srs3] ← TestMachine_001 (28 signals)
│ │ ├─ filler-02 [sidel SF300 Srs5] ← TestMachine_002 (28 signals)
│ │ ├─ capper-03 [khs Innofill Srs4] ← TestMachine_003 (28 signals)
│ │ ├─ labeler-04 [krones Contiroll Srs3] ← TestMachine_004 (28 signals)
│ │ ├─ inspector-05 [antares-vision Vmax Srs2] ← TestMachine_005 (28 signals)
│ │ ├─ coder-06 [videojet 1580 Srs4] ← TestMachine_006 (28 signals)
│ │ └─ rinser-07 [krones Hydra Srs3] ← TestMachine_007 (28 signals)
│ ├─ line-2/
│ │ ├─ rinser-08 [krones Hydra Srs3] ← TestMachine_008 (28 signals)
│ │ ├─ filler-09 [sidel SF300 Srs2] ← TestMachine_009 (28 signals)
│ │ ├─ capper-10 [khs Innofill Srs3] ← TestMachine_010 (28 signals)
│ │ ├─ labeler-11 [krones Contiroll Srs4] ← TestMachine_011 (28 signals)
│ │ ├─ inspector-12 [antares-vision Vmax Srs5] ← TestMachine_012 (28 signals)
│ │ └─ coder-13 [videojet 1580 Srs4] ← TestMachine_013 (28 signals)
│ └─ line-3/
│ ├─ rinser-14 [krones Hydra Srs4] ← TestMachine_014 (28 signals)
│ ├─ filler-15 [sidel SF300 Srs4] ← TestMachine_015 (28 signals)
│ ├─ capper-16 [khs Innofill Srs4] ← TestMachine_016 (28 signals)
│ ├─ labeler-17 [krones Contiroll Srs4] ← TestMachine_017 (28 signals)
│ ├─ inspector-18 [antares-vision Vmax Srs4] ← TestMachine_018 (28 signals)
│ └─ coder-19 [videojet 1580 Srs5] ← TestMachine_019 (28 signals)
├─ blending/ (Blending & CIP; from Galaxy TestArea2)
│ └─ cip-1/
│ └─ blender-20 [spx-flow APV-R5 Srs4] ← TestMachine_020 (24 signals)
└─ packaging/ (Packaging & Palletizing; from Galaxy TestArea3)
├─ pack-1/
│ ├─ cartoner-21 [marchesini MC820 Srs2] ← TestMachine_021 (24 signals)
│ ├─ case-packer-22 [bosch Elematic Srs4] ← TestMachine_022 (24 signals)
│ ├─ palletizer-23 [fanuc M410 Srs5] ← TestMachine_023 (24 signals)
│ ├─ stretch-wrapper-24 [lantech Q300 Srs4] ← TestMachine_024 (24 signals)
│ └─ checkweigher-25 [mettler-toledo C3570 Srs2] ← TestMachine_025 (24 signals)
├─ pack-2/
│ ├─ cartoner-26 [marchesini MC820 Srs2] ← TestMachine_026 (24 signals)
│ ├─ case-packer-27 [bosch Elematic Srs5] ← TestMachine_027 (24 signals)
│ ├─ palletizer-28 [fanuc M410 Srs5] ← TestMachine_028 (24 signals)
│ ├─ stretch-wrapper-29 [lantech Q300 Srs4] ← TestMachine_029 (24 signals)
│ └─ checkweigher-30 [mettler-toledo C3570 Srs5] ← TestMachine_030 (24 signals)
├─ pack-3/
│ ├─ cartoner-31 [marchesini MC820 Srs5] ← TestMachine_031 (24 signals)
│ ├─ case-packer-32 [bosch Elematic Srs5] ← TestMachine_032 (24 signals)
│ ├─ palletizer-33 [fanuc M410 Srs5] ← TestMachine_033 (24 signals)
│ ├─ stretch-wrapper-34 [lantech Q300 Srs4] ← TestMachine_034 (24 signals)
│ └─ checkweigher-35 [mettler-toledo C3570 Srs2] ← TestMachine_035 (24 signals)
└─ pack-4/
├─ cartoner-36 [marchesini MC820 Srs4] ← TestMachine_036 (24 signals)
├─ case-packer-37 [bosch Elematic Srs3] ← TestMachine_037 (24 signals)
├─ palletizer-38 [fanuc M410 Srs3] ← TestMachine_038 (24 signals)
├─ stretch-wrapper-39 [lantech Q300 Srs2] ← TestMachine_039 (24 signals)
└─ checkweigher-40 [mettler-toledo C3570 Srs5] ← TestMachine_040 (24 signals)
+114605
View File
File diff suppressed because it is too large Load Diff
+324
View File
@@ -0,0 +1,324 @@
#!/usr/bin/env python3
"""
Generate a fake company's Unified Namespace (UNS) for OtOpcUa, grounded in the
REAL AVEVA Galaxy "DEV" hierarchy pulled from the MxAccess gateway
(galaxy-hierarchy.json).
Mapping onto OtOpcUa's config-DB model (ServerCluster -> Namespace(Kind=Equipment)
-> UnsArea -> UnsLine -> Equipment -> Tag), with the canonical UNS path:
Enterprise / Site / Area / Line / Equipment / Signal
Real grounding:
* The 40 $TestMachine instances (TestMachine_001..040) become Equipment ("machines").
* The Galaxy areas they are deployed in (TestArea / TestArea2 / TestArea3)
become UNS Areas.
* Each machine's real DELMIA + MES receiver children (resolved from Galaxy
containment) become tag sub-folders ("delmia" / "mes") whose Tags carry the
REAL MXAccess fullTagReference.
* Process Tags map the real $TestMachine UDAs (TestChangingInt, TestDouble, ...)
to believable signal names while preserving the real reference.
Synthesized ("fake company") parts: enterprise/site/area/line names, machine
archetypes, and asset metadata (manufacturer/model/serial/SAP id). Everything
synthetic is clearly under generated identifiers; every node keeps a `source`
block linking back to the real Galaxy object.
OtOpcUa UNS-segment rule enforced: Area/Line/Equipment names match ^[a-z0-9-]{1,32}$.
"""
import json, re, collections, hashlib
SRC = "galaxy-hierarchy.json"
SEG = re.compile(r"^[a-z0-9-]{1,32}$")
def seg(s):
assert SEG.match(s), f"invalid UNS segment: {s!r}"
return s
# ---------------------------------------------------------------- load source
gal = json.load(open(SRC))
objs = gal["objects"]
byid = {o["gobjectId"]: o for o in objs}
def name(i): return byid[i]["tagName"] if i in byid else None
def children(pid): return [o for o in objs if o.get("parentGobjectId") == pid]
def chain_has(o, frag):
return any(frag in c for c in o.get("templateChain", []))
# the 40 top-level machines, grouped by their Galaxy area
AREAS_GAL = {} # galaxy area tag -> list of machine objects (sorted by number)
for o in objs:
if "$TestMachine" in o.get("templateChain", []): # exactly $TestMachine (not .DelmiaReceiver etc.)
area = name(o.get("parentGobjectId"))
AREAS_GAL.setdefault(area, []).append(o)
for a in AREAS_GAL:
AREAS_GAL[a].sort(key=lambda o: o["tagName"])
# resolve each machine's real DELMIA / MES receiver child tag names
def receiver_refs(machine):
delmia = mes = None
for c in children(machine["gobjectId"]):
if chain_has(c, "DelmiaReceiver"): delmia = c
elif chain_has(c, "MESReceiver"): mes = c
return delmia, mes
# ---------------------------------------------------------------- fake company
ENTERPRISE = "northwind" # fictional consumer-goods manufacturer
SITE = "birmingham" # fictional plant (maps to Galaxy node/galaxy DEV)
GALAXY_NODE = "DESKTOP-6JL3KKO"
GALAXY = "DEV"
# Galaxy area -> (uns area segment, friendly label, line plan)
# line plan: list of (line-segment, count) consuming machines in order.
AREA_PLAN = {
"TestArea": ("filling", "Filling & Capping", [("line-1", 7), ("line-2", 6), ("line-3", 6)]),
"TestArea2": ("blending", "Blending & CIP", [("cip-1", 1)]),
"TestArea3": ("packaging", "Packaging & Palletizing",
[("pack-1", 5), ("pack-2", 5), ("pack-3", 5), ("pack-4", 5)]),
}
# believable machine archetypes cycled within a line, per area
ARCHETYPES = {
"filling": [("rinser","krones","Hydra"),("filler","sidel","SF300"),
("capper","khs","Innofill"),("labeler","krones","Contiroll"),
("inspector","antares-vision","Vmax"),("coder","videojet","1580")],
"blending": [("blender","spx-flow","APV-R5")],
"packaging": [("cartoner","marchesini","MC820"),("case-packer","bosch","Elematic"),
("palletizer","fanuc","M410"),("stretch-wrapper","lantech","Q300"),
("checkweigher","mettler-toledo","C3570")],
}
def asset(area_seg, archetype, model_series, gtag, gidx):
mfr_model = {a[0]:(a[1],a[2]) for a in ARCHETYPES[area_seg]}
mfr, model = mfr_model[archetype]
h = int(hashlib.sha1(gtag.encode()).hexdigest(), 16)
return {
"manufacturer": mfr,
"model": f"{model} Srs{2 + h % 4}",
"serialNumber": f"SN26{gidx:05d}",
"sapId": f"100{200000 + h % 700000}",
"zTag": f"Z{area_seg[:3].upper()}{gidx:04d}",
"hardwareRevision": f"H{1 + h % 4}.{h % 10}",
"softwareRevision": f"S{2 + h % 3}.{(h>>4) % 10}",
"equipmentClassRef": f"urn:northwind:equipclass:{archetype}",
}
# ----------------------------------------------------- signal (Tag) templates
def dtype(galaxy_type):
return {
"Boolean":"Boolean","Integer":"Int32","Double":"Double","Float":"Float",
"String":"String","Time":"DateTime","ElapsedTime":"Double",
"InternationalizedString":"String",
}.get(galaxy_type, "String")
# process signals: (uns tag name, real $TestMachine UDA, galaxy type, access, flags)
PROCESS_SIGNALS = [
("speed-rpm", "TestChangingInt", "Integer", "ReadOnly", {}),
("production-count", "TestHistoryValue","Integer", "ReadOnly", {"historize": True}),
("temperature-c", "TestDouble", "Double", "ReadOnly", {}),
("pressure-bar", "TestFloat", "Float", "ReadOnly", {}),
("cycle-time-s", "TestDuration", "ElapsedTime","ReadOnly",{}),
("last-cycle-ts", "TestDateTime", "Time", "ReadOnly", {}),
("safety-interlock", "ProtectedValue", "Boolean", "ReadWrite", {"secured": True}),
("maint-lockout", "ProtectedValue1", "Boolean", "ReadWrite", {"secured": True}),
("motor-fault", "TestAlarm001", "Boolean", "ReadOnly", {"alarm": True}),
("over-temp", "TestAlarm002", "Boolean", "ReadOnly", {"alarm": True}),
("jam-detected", "TestAlarm003", "Boolean", "ReadOnly", {"alarm": True}),
("in-alarm", "InAlarm", "Boolean", "ReadOnly", {}),
]
# DELMIA receiver signals (real $DelmiaReceiver attributes)
DELMIA_SIGNALS = [
("work-order", "WorkOrderNumber", "String", "ReadWrite", {}),
("part-number", "PartNumber", "String", "ReadWrite", {}),
("job-step", "JobStepNumber", "String", "ReadWrite", {}),
("recipe-path", "DownloadPath", "String", "ReadWrite", {}),
("recipe-dl-req", "RecipeDownloadFlag","Boolean","ReadWrite", {}),
("recipe-done", "RecipeProcessedFlag","Boolean","ReadOnly", {}),
("recipe-result", "RecipeProcessResult","Boolean","ReadOnly", {}),
("ready", "ReadyFlag", "Boolean","ReadOnly", {}),
]
# MES receiver signals (real $MESReceiver attributes)
MES_SIGNALS = [
("move-in-req", "MoveInFlag", "Boolean","ReadWrite", {}),
("move-in-batch", "MoveInBatchID", "Integer","ReadWrite", {}),
("move-in-operator","MoveInOperatorName", "String", "ReadOnly", {}),
("move-in-ok", "MoveInSuccessFlag", "Boolean","ReadOnly", {}),
("move-out-req", "MoveOutFlag", "Boolean","ReadWrite", {}),
("move-out-batch", "MoveOutBatchID", "Integer","ReadWrite", {}),
("move-out-ok", "MoveOutSuccessfulFlag","Boolean","ReadOnly", {}),
("container-id", "MoveOutMesContainerNum","String","ReadOnly", {}),
]
def attr_set(obj):
return {a["attributeName"] for a in obj["attributes"]} if obj else set()
def make_tags(equipment_id, machine, delmia, mes):
tags = []
def add(folder, signals, ref_obj):
if ref_obj is None:
return
ref_tag = ref_obj["tagName"]
have = attr_set(ref_obj)
for uns_name, attr, gtype, access, flags in signals:
if attr not in have:
continue # only emit signals whose real attribute exists on THIS instance
full = f"{ref_tag}.{attr}"
tid = f"tag-{equipment_id}-{(folder+'-' if folder else '')}{uns_name}"
t = {
"tagId": tid,
"name": uns_name,
"folderPath": folder,
"dataType": dtype(gtype),
"accessLevel": access,
"historize": bool(flags.get("historize", False)),
"tagConfig": {
"isAlarm": bool(flags.get("alarm", False)),
"secured": bool(flags.get("secured", False)),
},
"source": {
"namespaceKind": "SystemPlatform",
"fullTagReference": full,
"galaxyType": gtype,
},
}
tags.append(t)
add("", PROCESS_SIGNALS, machine)
add("delmia", DELMIA_SIGNALS, delmia)
add("mes", MES_SIGNALS, mes)
return tags
# ---------------------------------------------------------------- build UNS
cluster_id = f"{ENTERPRISE}-{SITE}"
uns_areas, uns_lines, equipment = [], [], []
src_hierarchy = {"galaxyNode": GALAXY_NODE, "galaxy": GALAXY, "areas": []}
gidx = 0
for gal_area, (area_seg, area_label, line_plan) in AREA_PLAN.items():
machines = AREAS_GAL.get(gal_area, [])
area_id = f"area-{area_seg}"
uns_areas.append({
"unsAreaId": area_id, "clusterId": cluster_id, "name": seg(area_seg),
"notes": area_label,
"source": {"galaxyArea": gal_area,
"galaxyGobjectId": next((o["gobjectId"] for o in objs
if o["tagName"] == gal_area), None)},
})
src_area = {"galaxyArea": gal_area, "machines": []}
# consume machines line-by-line per the plan
cursor = 0
for line_seg, count in line_plan:
line_id = f"line-{area_seg}-{line_seg}"
uns_lines.append({
"unsLineId": line_id, "unsAreaId": area_id, "name": seg(line_seg),
"notes": f"{area_label} {line_seg}",
})
archetypes = ARCHETYPES[area_seg]
for j in range(count):
if cursor >= len(machines): break
m = machines[cursor]; cursor += 1; gidx += 1
archetype, _, model_series = archetypes[j % len(archetypes)]
eq_seg = seg(f"{archetype}-{gidx:02d}")
eq_id = f"eq-{area_seg}-{line_seg}-{eq_seg}"
delmia, mes = receiver_refs(m)
a = asset(area_seg, archetype, model_series, m["tagName"], gidx)
tags = make_tags(eq_id, m, delmia, mes)
equipment.append({
"equipmentId": eq_id, "unsLineId": line_id, "name": eq_seg,
"machineCode": m["tagName"],
"manufacturer": a["manufacturer"], "model": a["model"],
"serialNumber": a["serialNumber"], "sapId": a["sapId"],
"zTag": a["zTag"], "hardwareRevision": a["hardwareRevision"],
"softwareRevision": a["softwareRevision"],
"assetLocation": f"{ENTERPRISE}/{SITE}/{area_seg}/{line_seg}",
"equipmentClassRef": a["equipmentClassRef"],
"unsPath": f"{ENTERPRISE}/{SITE}/{area_seg}/{line_seg}/{eq_seg}",
"source": {
"namespaceKind": "SystemPlatform",
"galaxyTag": m["tagName"], "galaxyGobjectId": m["gobjectId"],
"templateChain": m["templateChain"], "galaxyArea": gal_area,
"delmiaReceiverTag": delmia["tagName"] if delmia else None,
"mesReceiverTag": mes["tagName"] if mes else None,
},
"tags": tags,
})
src_area["machines"].append({
"tag": m["tagName"], "gobjectId": m["gobjectId"],
"unsEquipment": f"{ENTERPRISE}/{SITE}/{area_seg}/{line_seg}/{eq_seg}",
"delmiaReceiver": delmia["tagName"] if delmia else None,
"mesReceiver": mes["tagName"] if mes else None,
})
src_hierarchy["areas"].append(src_area)
tag_count = sum(len(e["tags"]) for e in equipment)
doc = {
"$schema": "internal://otopcua/uns-export/v1",
"kind": "fake-company-uns",
"description": ("Fictional company UNS generated around the real AVEVA Galaxy "
"'DEV' TestMachine instances and the areas they are deployed in. "
"Modeled on OtOpcUa's Cluster->Namespace->UnsArea->UnsLine->"
"Equipment->Tag schema. Synthetic naming/asset metadata; every "
"node links back to its real Galaxy source."),
"generatedFrom": {
"file": SRC, "galaxyNode": GALAXY_NODE, "galaxy": GALAXY,
"gatewayEndpoint": "http://10.100.0.48:5120 (MxAccess gateway, gRPC)",
"sourceObjectCount": len(objs),
},
"company": {"name": "Northwind Consumer Products",
"enterprise": ENTERPRISE, "site": SITE,
"siteDescription": "Birmingham bottling & packaging plant"},
"cluster": {"clusterId": cluster_id, "name": f"{ENTERPRISE}-{SITE}-uns",
"enterprise": ENTERPRISE, "site": SITE,
"redundancyMode": "Hot", "nodeCount": 2},
"namespace": {"namespaceId": f"{cluster_id}-equipment", "clusterId": cluster_id,
"kind": "Equipment",
"namespaceUri": f"urn:{ENTERPRISE}:{SITE}:uns"},
"uns": {"unsAreas": uns_areas, "unsLines": uns_lines, "equipment": equipment},
"stats": {"areas": len(uns_areas), "lines": len(uns_lines),
"equipment": len(equipment), "tags": tag_count},
"sourceGalaxyHierarchy": src_hierarchy,
}
json.dump(doc, open("company-uns.json", "w"), indent=2)
# ---------------------------------------------------------------- tree view
lines = []
lines.append(f"Northwind Consumer Products — Unified Namespace")
lines.append(f"(generated from Galaxy {GALAXY_NODE}\\{GALAXY}; "
f"{len(equipment)} machines, {tag_count} signals)")
lines.append("")
lines.append(f"{ENTERPRISE}")
lines.append(f"└─ {SITE}")
area_by_id = {a["unsAreaId"]: a for a in uns_areas}
lines_by_area = collections.defaultdict(list)
for l in uns_lines: lines_by_area[l["unsAreaId"]].append(l)
eq_by_line = collections.defaultdict(list)
for e in equipment: eq_by_line[e["unsLineId"]].append(e)
for ai, a in enumerate(uns_areas):
a_last = ai == len(uns_areas) - 1
a_pre = " " + ("└─ " if a_last else "├─ ")
lines.append(f" {'└─' if a_last else '├─'} {a['name']}/ ({a['notes']}; from Galaxy {a['source']['galaxyArea']})")
als = lines_by_area[a["unsAreaId"]]
a_cont = " " if a_last else ""
for li, l in enumerate(als):
l_last = li == len(als) - 1
lines.append(f"{a_cont}{'└─' if l_last else '├─'} {l['name']}/")
eqs = eq_by_line[l["unsLineId"]]
l_cont = a_cont + (" " if l_last else "")
for ei, e in enumerate(eqs):
e_last = ei == len(eqs) - 1
ntags = len(e["tags"])
lines.append(f"{l_cont}{'└─' if e_last else '├─'} {e['name']} "
f"[{e['manufacturer']} {e['model']}] ← {e['source']['galaxyTag']} ({ntags} signals)")
open("company-uns.tree.txt", "w").write("\n".join(lines) + "\n")
print(json.dumps(doc["stats"], indent=2))
print("\nsample equipment:")
e = equipment[0]
print(f" {e['unsPath']} machineCode={e['machineCode']} delmia={e['source']['delmiaReceiverTag']} mes={e['source']['mesReceiverTag']}")
print(" sample tags:")
for t in e["tags"][:3] + [e["tags"][12]] + [e["tags"][-1]]:
print(f" {t['folderPath'] or '.'}/{t['name']:18s} {t['dataType']:8s} {t['accessLevel']:9s} -> {t['source']['fullTagReference']}")
print("\nwrote: company-uns.json, company-uns.tree.txt")
+7
View File
@@ -0,0 +1,7 @@
# Python virtualenv + caches (recreate via: python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txt)
.venv/
__pycache__/
*.pyc
# Generated by `otopcua_uns.py generate` from ../galaxy-hierarchy.json
load-plan.json
+121
View File
@@ -0,0 +1,121 @@
# otopcua-uns-loader
A **reloadable** populate-and-verify tool for the OtOpcUa galaxy Unified Namespace.
Recreates a UNS load grounded in the real AVEVA Galaxy **DEV** hierarchy (the 40
`TestMachine` instances) and verifies it streams **live values** on OPC UA — so
you can rebuild the OtOpcUa docker-dev instance and get the namespace back with
one populate + one deploy click.
## What it loads
One **SystemPlatform** `Tag` per `(machine, signal)` bound to the existing
`GalaxyMxGateway` driver (`MAIN-galaxy-mxgw`). Each tag's `FolderPath` is the
Galaxy object and its `Name` the attribute, so the materialised OPC UA variable
`OtOpcUa/<machine>/<signal>` has a NodeId equal to the MXAccess reference the
driver subscribes to — which is what makes the value go live.
Signals mirrored per machine (only those the instance actually has): the
`$TestMachine` process UDAs — `TestChangingInt`, `TestHistoryValue`,
`TestDouble/Float/Duration/DateTime`, `ProtectedValue(1)`, `TestAlarm001..003`,
`InAlarm`**396 tags across 40 machines**.
Every row carries the `nw-mirror-` `TagId` prefix, so `clean` removes exactly
what the tool created (adopting any pre-existing seed row for the same ref).
## Why a deploy click is in the middle
OtOpcUa applies config only from **sealed Deployment snapshots**, and the only
way to seal one is the AdminUI **"Deploy current configuration"** button
(`http://localhost:9200/deployments`) — there is no SQL/REST/CLI trigger (it's
an in-cluster Akka operation). So the flow is:
```
populate ──SQL──▶ live config tables
│ (you click Deploy at :9200, sign in multi-role/password)
driver applies ▶ materialises variables ▶ SubscribeBulk ▶ live values
verify ──OPC UA──▶ browse + read Good values on :4840
```
`populate` and `clean` print the reminder; `verify --wait` polls until the
deploy lands.
> Live values depend on the driver **SubscribeBulk** pass
> (OtOpcUa `master` ≥ commit `c1ce583`). On older builds variables materialise
> but stay `BadWaitingForInitialData`.
## Setup
```bash
cd otopcua-uns-loader
python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt
```
## Use
```bash
./.venv/bin/python otopcua_uns.py generate # build load-plan.json from galaxy-hierarchy.json
./.venv/bin/python otopcua_uns.py populate # upsert the 396 mirror Tag rows (idempotent)
# → open http://localhost:9200/deployments, sign in, click "Deploy current configuration"
./.venv/bin/python otopcua_uns.py verify --wait # poll until live values are Good on :4840
./.venv/bin/python otopcua_uns.py status # config-DB + address-space snapshot
./.venv/bin/python otopcua_uns.py clean # remove all nw-mirror-* tags (then Deploy again)
```
### Rebuild recovery (the point of the tool)
After the docker-dev instance is rebuilt (DB wiped):
1. Ensure the schema + clusters + the `MAIN-galaxy-mxgw` driver exist
(`dotnet ef database update` + the docker-dev `cluster-seed`; see the OtOpcUa
`docker-dev/README.md`).
2. `populate`**Deploy** at the AdminUI → `verify --wait`.
## Troubleshooting
**`verify` stays INCOMPLETE / deployment "Sealed" but drivers never applied it.**
If you recreate the **admin/coordinator** node (`admin-a`) around the same time
you click Deploy, the dispatch broadcast can be lost — the deployment seals but
`NodeDeploymentState` shows no row for the driver nodes, and the address space
keeps the old content. Recover by: restart the driver nodes
(`docker restart otopcua-dev-driver-a-1 otopcua-dev-driver-b-1`) so they cleanly
re-subscribe, then Deploy again. If a no-op "NoChanges" blocks the re-deploy,
delete the orphan sealed `Deployment` row that no node applied (it has no
`NodeDeploymentState` children) so Deploy sees drift again.
**A rebuilt/restarted node serves an empty address space until the next Deploy.**
On bootstrap a node recovers to its last-*applied* revision and does **not**
re-materialise until a new deployment is dispatched — so after any node restart,
click Deploy once (a config change bumps the revision) to repopulate + re-subscribe.
## Configuration
Defaults target docker-dev; override via flags or env:
| Flag | Env | Default |
|---|---|---|
| `--sql-host/-port/-user/-password/-db` | `OTOPCUA_SQL_*` | `localhost:14330` sa / `OtOpcUa!Dev123` / `OtOpcUa` |
| `--opcua-endpoint` | `OTOPCUA_OPCUA_ENDPOINT` | `opc.tcp://localhost:4840` |
| `--driver` | `OTOPCUA_GALAXY_DRIVER` | `MAIN-galaxy-mxgw` |
| `--galaxy-json` | `OTOPCUA_GALAXY_JSON` | `../galaxy-hierarchy.json` |
| `--deploy-url` | — | `http://localhost:9200/deployments` |
## Files
- `otopcua_uns.py` — the CLI (generate / populate / verify / status / clean)
- `load-plan.json` — generated load plan (machine → signal → MXAccess ref)
- `../galaxy-hierarchy.json` — the source of truth, pulled live from the gateway
- `requirements.txt`, `.venv/`
## Scope note — company-UNS shape
This tool loads the galaxy in its **native hierarchy**
(`OtOpcUa/TestMachine_NNN/<signal>`), which is the only shape that can carry live
Galaxy values: OtOpcUa forbids the `GalaxyMxGateway` driver in an `Equipment`
namespace, so a custom `Enterprise/Site/Area/Line/Equipment` UNS (e.g. the
Northwind model in `../company-uns.json`) must be a separate **Equipment**
namespace fed by an `OpcUaClient` driver + `UnsMappingTable` that remaps this
mirror. That overlay is the designed next layer; `../company-uns.json` already
carries the area/line/equipment → galaxy-ref mapping it needs.
+332
View File
@@ -0,0 +1,332 @@
#!/usr/bin/env python3
"""
otopcua_uns.py — reloadable populate + verify for the OtOpcUa galaxy UNS.
Recreates and verifies an OtOpcUa Unified-Namespace load grounded in the real
AVEVA Galaxy "DEV" hierarchy (the 40 TestMachine instances). Designed to be
re-run after the OtOpcUa docker-dev instance is rebuilt.
Pipeline (see scadaproj/memory otopcua-uns-deploy-and-value-streaming):
populate ──SQL──▶ live config tables (Tag rows, nw-* prefix)
you click "Deploy current configuration" at :9200
driver applies ▶ materialises OtOpcUa/<machine>/<signal> ▶ SubscribeBulk
verify ──OPC UA──▶ browse + read live values on :4840
What it loads: one SystemPlatform Tag per (machine, signal) bound to the
existing GalaxyMxGateway driver. Each tag's FolderPath is the Galaxy object and
its Name the attribute, so the materialised variable NodeId is exactly the
MXAccess ref the driver subscribes to — giving live values. Every row carries
the `nw-` id prefix so `clean` can remove them without touching other config.
Idempotent: populate upserts by TagId; re-running is a no-op when unchanged.
Subcommands:
generate Build the load plan from galaxy-hierarchy.json (writes load-plan.json)
populate Upsert the SystemPlatform mirror Tag rows into the config DB
verify Check DB rows present + live OPC UA values are Good on :4840
status Show config-DB + address-space state
clean Delete all nw-* mirror Tag rows
Deploy is a human-gated AdminUI action (no SQL/REST trigger exists); populate
and clean print the reminder and `verify --wait` polls until it lands.
Deps: pymssql, asyncua (see requirements.txt; use the bundled .venv).
"""
import argparse
import json
import os
import re
import sys
import time
# ── config (overridable via env / flags) ───────────────────────────────────
DEF_MSSQL = dict(
host=os.environ.get("OTOPCUA_SQL_HOST", "localhost"),
port=int(os.environ.get("OTOPCUA_SQL_PORT", "14330")),
user=os.environ.get("OTOPCUA_SQL_USER", "sa"),
password=os.environ.get("OTOPCUA_SQL_PASSWORD", "OtOpcUa!Dev123"),
database=os.environ.get("OTOPCUA_SQL_DB", "OtOpcUa"),
)
DEF_OPCUA = os.environ.get("OTOPCUA_OPCUA_ENDPOINT", "opc.tcp://localhost:4840")
DEF_DRIVER = os.environ.get("OTOPCUA_GALAXY_DRIVER", "MAIN-galaxy-mxgw")
DEF_GALAXY_JSON = os.environ.get(
"OTOPCUA_GALAXY_JSON",
os.path.join(os.path.dirname(__file__), "..", "galaxy-hierarchy.json"),
)
ID_PREFIX = "nw-mirror-" # all rows we own carry this TagId prefix
LOAD_PLAN = os.path.join(os.path.dirname(__file__), "load-plan.json")
# ── the value signals we mirror, per $TestMachine instance ──────────────────
# (galaxy attribute name, OtOpcUa DriverDataType, access '0'=Read/'1'=ReadWrite)
SIGNALS = [
("TestChangingInt", "Int32", "0"),
("TestHistoryValue", "Int32", "0"),
("TestDouble", "Float64", "0"),
("TestFloat", "Float32", "0"),
("TestDuration", "Float64", "0"),
("TestDateTime", "DateTime", "0"),
("ProtectedValue", "Boolean", "1"),
("ProtectedValue1", "Boolean", "1"),
("TestAlarm001", "Boolean", "0"),
("TestAlarm002", "Boolean", "0"),
("TestAlarm003", "Boolean", "0"),
("InAlarm", "Boolean", "0"),
]
# ── plan generation (grounded in the real galaxy) ───────────────────────────
def build_plan(galaxy_json, driver):
with open(galaxy_json) as f:
gal = json.load(f)
machines = [
o for o in gal["objects"]
if "$TestMachine" in o.get("templateChain", [])
]
machines.sort(key=lambda o: o["tagName"])
rows = []
for m in machines:
have = {a["attributeName"] for a in m["attributes"]}
for attr, dtype, access in SIGNALS:
if attr not in have:
continue # only mirror attributes this instance really has
rows.append({
"tag_id": f"{ID_PREFIX}{m['tagName']}-{attr}".lower(),
"driver_instance_id": driver,
"name": attr,
"folder_path": m["tagName"], # → folder; ref = folder.name
"data_type": dtype,
"access_level": access,
"mxaccess_ref": f"{m['tagName']}.{attr}",
})
return {
"source": galaxy_json,
"driver_instance_id": driver,
"machines": len(machines),
"tags": len(rows),
"rows": rows,
}
# ── DB helpers ──────────────────────────────────────────────────────────────
def connect(cfg):
import pymssql
conn = pymssql.connect(
server=cfg["host"], port=str(cfg["port"]), user=cfg["user"],
password=cfg["password"], database=cfg["database"], autocommit=False,
)
cur = conn.cursor()
# The Tag table has filtered indexes / computed columns; writes require this.
cur.execute("SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON;")
return conn, cur
def driver_exists(cur, driver):
cur.execute(
"SELECT n.Kind FROM dbo.DriverInstance d "
"JOIN dbo.Namespace n ON n.NamespaceId = d.NamespaceId "
"WHERE d.DriverInstanceId = %s", (driver,))
r = cur.fetchone()
return r[0] if r else None
# ── commands ────────────────────────────────────────────────────────────────
def cmd_generate(args):
plan = build_plan(args.galaxy_json, args.driver)
with open(LOAD_PLAN, "w") as f:
json.dump(plan, f, indent=2)
print(f"plan: {plan['machines']} machines → {plan['tags']} mirror tags (driver {plan['driver_instance_id']})")
print(f"wrote {LOAD_PLAN}")
return 0
def cmd_populate(args):
plan = build_plan(args.galaxy_json, args.driver)
conn, cur = connect(args.mssql)
kind = driver_exists(cur, args.driver)
if kind is None:
print(f"ERROR: driver instance '{args.driver}' not found in config DB.", file=sys.stderr)
return 2
if kind != "SystemPlatform":
print(f"ERROR: driver '{args.driver}' is in a {kind} namespace; the galaxy mirror needs a "
f"SystemPlatform/GalaxyMxGateway driver.", file=sys.stderr)
return 2
inserted = updated = 0
for r in plan["rows"]:
# Upsert by the SystemPlatform natural key (DriverInstanceId, FolderPath, Name)
# — the UX_Tag_FolderPath unique index. This adopts any pre-existing seed row for
# the same ref into our nw-* set (so `clean` can remove it) and stays idempotent on
# re-run. RowId/RowVersion are server-managed.
cur.execute(
"SELECT TagId FROM dbo.Tag WHERE DriverInstanceId=%s AND FolderPath=%s "
"AND Name=%s AND EquipmentId IS NULL",
(r["driver_instance_id"], r["folder_path"], r["name"]))
if cur.fetchone():
cur.execute(
"UPDATE dbo.Tag SET TagId=%s, DataType=%s, AccessLevel=%s, "
"WriteIdempotent=0, TagConfig='{}' "
"WHERE DriverInstanceId=%s AND FolderPath=%s AND Name=%s AND EquipmentId IS NULL",
(r["tag_id"], r["data_type"], r["access_level"],
r["driver_instance_id"], r["folder_path"], r["name"]))
updated += 1
else:
cur.execute(
"INSERT INTO dbo.Tag (TagRowId, TagId, DriverInstanceId, EquipmentId, "
"Name, FolderPath, DataType, AccessLevel, WriteIdempotent, TagConfig) "
"VALUES (NEWID(), %s, %s, NULL, %s, %s, %s, %s, 0, '{}')",
(r["tag_id"], r["driver_instance_id"], r["name"], r["folder_path"],
r["data_type"], r["access_level"]))
inserted += 1
conn.commit()
conn.close()
print(f"populated: {inserted} inserted, {updated} updated "
f"({plan['tags']} mirror tags across {plan['machines']} machines)")
print()
print(">>> NEXT: open the AdminUI, sign in, and click "
"'Deploy current configuration' to seal + serve the load:")
print(f" {args.deploy_url}")
print(">>> then run: otopcua_uns.py verify --wait")
return 0
def cmd_clean(args):
conn, cur = connect(args.mssql)
cur.execute("DELETE FROM dbo.Tag WHERE TagId LIKE %s", (ID_PREFIX + "%",))
n = cur.rowcount
conn.commit()
conn.close()
print(f"removed {n} nw-* mirror tag(s). Deploy again at {args.deploy_url} to drop them from the address space.")
return 0
def cmd_status(args):
conn, cur = connect(args.mssql)
cur.execute("SELECT COUNT(*) FROM dbo.Tag WHERE TagId LIKE %s", (ID_PREFIX + "%",))
db_tags = cur.fetchone()[0]
# Deployment.Status: 2 = Sealed (the snapshot driver nodes apply).
cur.execute("SELECT TOP 1 RevisionHash, SealedAtUtc FROM dbo.Deployment "
"WHERE Status=2 ORDER BY SealedAtUtc DESC")
dep = cur.fetchone()
conn.close()
print(f"config DB : {db_tags} mirror tags (nw-*) present")
print(f"last sealed : {('rev '+dep[0][:12]+'… @ '+str(dep[1])) if dep else '(none)'}")
folders, variables, good = browse_summary(args.opcua_endpoint)
print(f"address space : {folders} machine folder(s), {variables} variable(s), {good} value(s) Good on {args.opcua_endpoint}")
return 0
def cmd_verify(args):
plan = build_plan(args.galaxy_json, args.driver)
expected = plan["tags"]
deadline = time.time() + (args.wait_seconds if args.wait else 0)
while True:
folders, variables, good = browse_summary(args.opcua_endpoint)
ok = variables >= expected and good >= max(1, int(expected * 0.5))
if ok or time.time() >= deadline:
break
print(f" waiting for deploy… ({variables}/{expected} vars, {good} Good)")
time.sleep(5)
print(f"expected mirror tags : {expected}")
print(f"address-space vars : {variables} (in {folders} folders)")
print(f"values Good (live) : {good}")
sample = sample_values(args.opcua_endpoint, 6)
for nm, val, sc in sample:
print(f" {nm} = {val} [{sc}]")
passed = variables >= expected and good >= max(1, int(expected * 0.5))
print("VERIFY:", "PASS — UNS loaded and live" if passed else "INCOMPLETE — did you Deploy at the AdminUI?")
return 0 if passed else 1
# ── OPC UA helpers (asyncua) ────────────────────────────────────────────────
def browse_summary(endpoint):
import asyncio
from asyncua import Client
async def run():
folders = variables = good = 0
async with Client(endpoint) as c:
for k in await c.nodes.objects.get_children():
if (await k.read_browse_name()).Name != "OtOpcUa":
continue
for f in await k.get_children():
folders += 1
for v in await f.get_children():
variables += 1
try:
dv = await v.read_data_value()
if dv.StatusCode and dv.StatusCode.is_good():
good += 1
except Exception:
pass
return folders, variables, good
try:
return asyncio.run(run())
except Exception as e:
return (f"<{type(e).__name__}>", 0, 0)
def sample_values(endpoint, n):
import asyncio
from asyncua import Client
async def run():
out = []
async with Client(endpoint) as c:
for k in await c.nodes.objects.get_children():
if (await k.read_browse_name()).Name != "OtOpcUa":
continue
for f in await k.get_children():
for v in await f.get_children():
try:
dv = await v.read_data_value()
sc = dv.StatusCode.name if dv.StatusCode else "?"
out.append(((await v.read_browse_name()).Name, dv.Value.Value, sc))
except Exception as e:
out.append(((await v.read_browse_name()).Name, f"<{type(e).__name__}>", "?"))
if len(out) >= n:
return out
return out
try:
return asyncio.run(run())
except Exception as e:
return [("<browse error>", str(e), "?")]
# ── arg parsing ─────────────────────────────────────────────────────────────
def main(argv):
p = argparse.ArgumentParser(description="Reloadable populate + verify for the OtOpcUa galaxy UNS.")
p.add_argument("--galaxy-json", default=DEF_GALAXY_JSON)
p.add_argument("--driver", default=DEF_DRIVER, help="SystemPlatform GalaxyMxGateway driver instance id")
p.add_argument("--opcua-endpoint", default=DEF_OPCUA)
p.add_argument("--deploy-url", default="http://localhost:9200/deployments")
p.add_argument("--sql-host", default=DEF_MSSQL["host"])
p.add_argument("--sql-port", type=int, default=DEF_MSSQL["port"])
p.add_argument("--sql-user", default=DEF_MSSQL["user"])
p.add_argument("--sql-password", default=DEF_MSSQL["password"])
p.add_argument("--sql-db", default=DEF_MSSQL["database"])
sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("generate")
sub.add_parser("populate")
sub.add_parser("clean")
sub.add_parser("status")
vp = sub.add_parser("verify")
vp.add_argument("--wait", action="store_true", help="poll until the deploy lands")
vp.add_argument("--wait-seconds", type=int, default=120)
a = p.parse_args(argv)
a.mssql = dict(host=a.sql_host, port=a.sql_port, user=a.sql_user,
password=a.sql_password, database=a.sql_db)
return {
"generate": cmd_generate, "populate": cmd_populate, "clean": cmd_clean,
"status": cmd_status, "verify": cmd_verify,
}[a.cmd](a)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+2
View File
@@ -0,0 +1,2 @@
pymssql>=2.3
asyncua>=2.0