dohertj2/3yearplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Expand SnowBridge to own ingest + in-process transform; drop dbt

Browse Source 
SnowBridge now owns machine-data ingest, in-process .NET transformation,
and direct writes to curated tables in Snowflake. Collapses the previous
ingest/transform split into a single service; no dbt, no external
orchestrator, no Snowflake landing tier. Keeps the in-house .NET pattern
consistent with ScadaBridge and OtOpcUa.

The "Snowflake dbt Transform Layer" roadmap workstream merges into
SnowBridge (7 → 6 workstreams); Year 2 canonical-state-based OEE moves
with it. Canonical model still has three surfaces — the third is
renamed from "dbt curated layer" to "SnowBridge curated layer in
Snowflake"; mechanics unchanged.
This commit is contained in:
Joseph Doherty
2026-04-24 14:53:16 -04:00
parent 22a86974f6
commit 98bf2d0da4
15 changed files with 130 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
schemas/CONTRIBUTING.md
View File

@@ -7,7 +7,7 @@
1. **Open an issue first** for any new equipment class, UNS subtree, or format change. Describe the use case + the consumer(s) that need it.
2. **Branch + PR** — work on a feature branch, open a PR against `main`.
3. **CI gate** validates every JSON file against the schema in `format/`.
4. **Review** — at least one schemas-repo maintainer + one consumer-team representative (OtOpcUa, Redpanda, or dbt depending on what changed).
4. **Review** — at least one schemas-repo maintainer + one consumer-team representative (OtOpcUa, Redpanda, or SnowBridge depending on what changed).
5. **Merge + tag** — merge to `main` and create a semver tag. Consumers pin to tags.
## Adding a new equipment class
@@ -30,7 +30,7 @@
Editing files in `format/` is a breaking change for downstream consumers. Process:
1. Open an issue with the proposed change + rationale.
2. Notify all consumer teams (OtOpcUa, Redpanda, dbt, anyone else listed in `docs/consumer-integration.md`).
2. Notify all consumer teams (OtOpcUa, Redpanda, SnowBridge, anyone else listed in `docs/consumer-integration.md`).
3. Get explicit signoff from each before merging.
4. Bump the major version of every affected class file simultaneously (consumers use this to detect breaking changes).

2
schemas/README.md
View File

@@ -20,7 +20,7 @@ Three surfaces per the 3-year-plan handoff §"Canonical Model Integration":
|----------|-----|
| **OtOpcUa equipment namespace** | At deploy/config time, OtOpcUa nodes fetch the equipment-class template referenced by `Equipment.EquipmentClassRef` and use it to validate the operator-configured tag set. Drift = config validation error |
| **Redpanda topics + Protobuf schemas** | Equipment-class templates derive Protobuf message definitions for canonical events (`equipment.state.transitioned`, etc.) |
| **dbt curated layer in Snowflake** | Same templates derive column definitions and dimension tables for the curated analytics model |
| **SnowBridge curated layer in Snowflake** | Same templates derive column definitions and dimension tables for the curated analytics model. SnowBridge owns ingest + in-process .NET transform + write; no separate dbt layer. |
OtOpcUa is one consumer of three. Decisions about format, structure, and naming live with the schemas-repo owner team (TBD), not with any one consumer.

12
schemas/docs/consumer-integration.md
View File

@@ -27,16 +27,16 @@ How each of the three canonical-model consumers integrates with this repo.
**Status (2026-04-17)**: not wired. Redpanda team to design the codegen step when the schemas repo has a stable initial class set.
## dbt curated layer in Snowflake
## SnowBridge curated layer in Snowflake
**What it pulls**: equipment-class templates derive column definitions for the curated equipment-state and equipment-signal models in dbt.
**What it pulls**: equipment-class templates derive column definitions for the curated equipment-state and equipment-signal tables SnowBridge writes into Snowflake. SnowBridge owns ingest + in-process transform + write — there is no separate dbt layer.
**Integration points**:
- A dbt macro reads `classes/*.json` and generates per-class staging models with the canonical signal columns.
- UNS subtree definitions (`uns/*.json`) drive the dim_site / dim_area / dim_line dimension tables.
- Versioning: dbt project pins to a specific schemas-repo tag; updates require an explicit dbt deploy.
- A SnowBridge codegen step reads `classes/*.json` and generates per-class transform definitions + curated-table DDL with the canonical signal columns.
- UNS subtree definitions (`uns/*.json`) drive the dim_site / dim_area / dim_line dimension tables written by SnowBridge.
- Versioning: the SnowBridge build pins to a specific schemas-repo tag; updates require an explicit SnowBridge release.
**Status (2026-04-17)**: not wired. dbt team to design the macro when the schemas repo has a stable initial class set.
**Status (2026-04-24)**: not wired. SnowBridge team to design the codegen step when the schemas repo has a stable initial class set.
## Cross-consumer compatibility

2
schemas/docs/overview.md
View File

@@ -12,7 +12,7 @@ Three OT/IT systems consume the same canonical model:
- **OtOpcUa** equipment namespace — exposes raw signals over OPC UA
- **Redpanda + Protobuf** event topics — canonical event shape on the wire
- **dbt curated layer in Snowflake** — analytics model
- **SnowBridge curated layer in Snowflake** — analytics model (SnowBridge writes curated rows directly; no separate dbt layer)
Without a central source, they would drift. With one repo, every consumer pulls a versioned snapshot and validates against it. Drift becomes a CI failure, not a production incident.