dohertj2/3yearplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Expand SnowBridge to own ingest + in-process transform; drop dbt

Browse Source 
SnowBridge now owns machine-data ingest, in-process .NET transformation,
and direct writes to curated tables in Snowflake. Collapses the previous
ingest/transform split into a single service; no dbt, no external
orchestrator, no Snowflake landing tier. Keeps the in-house .NET pattern
consistent with ScadaBridge and OtOpcUa.

The "Snowflake dbt Transform Layer" roadmap workstream merges into
SnowBridge (7 → 6 workstreams); Year 2 canonical-state-based OEE moves
with it. Canonical model still has three surfaces — the third is
renamed from "dbt curated layer" to "SnowBridge curated layer in
Snowflake"; mechanics unchanged.
This commit is contained in:
Joseph Doherty
2026-04-24 14:53:16 -04:00
parent 22a86974f6
commit 98bf2d0da4
15 changed files with 130 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
schemas/docs/consumer-integration.md
View File

@@ -27,16 +27,16 @@ How each of the three canonical-model consumers integrates with this repo.
**Status (2026-04-17)**: not wired. Redpanda team to design the codegen step when the schemas repo has a stable initial class set.
## dbt curated layer in Snowflake
## SnowBridge curated layer in Snowflake
**What it pulls**: equipment-class templates derive column definitions for the curated equipment-state and equipment-signal models in dbt.
**What it pulls**: equipment-class templates derive column definitions for the curated equipment-state and equipment-signal tables SnowBridge writes into Snowflake. SnowBridge owns ingest + in-process transform + write — there is no separate dbt layer.
**Integration points**:
- A dbt macro reads `classes/*.json` and generates per-class staging models with the canonical signal columns.
- UNS subtree definitions (`uns/*.json`) drive the dim_site / dim_area / dim_line dimension tables.
- Versioning: dbt project pins to a specific schemas-repo tag; updates require an explicit dbt deploy.
- A SnowBridge codegen step reads `classes/*.json` and generates per-class transform definitions + curated-table DDL with the canonical signal columns.
- UNS subtree definitions (`uns/*.json`) drive the dim_site / dim_area / dim_line dimension tables written by SnowBridge.
- Versioning: the SnowBridge build pins to a specific schemas-repo tag; updates require an explicit SnowBridge release.
**Status (2026-04-17)**: not wired. dbt team to design the macro when the schemas repo has a stable initial class set.
**Status (2026-04-24)**: not wired. SnowBridge team to design the codegen step when the schemas repo has a stable initial class set.
## Cross-consumer compatibility

2
schemas/docs/overview.md
View File

@@ -12,7 +12,7 @@ Three OT/IT systems consume the same canonical model:
- **OtOpcUa** equipment namespace — exposes raw signals over OPC UA
- **Redpanda + Protobuf** event topics — canonical event shape on the wire
- **dbt curated layer in Snowflake** — analytics model
- **SnowBridge curated layer in Snowflake** — analytics model (SnowBridge writes curated rows directly; no separate dbt layer)
Without a central source, they would drift. With one repo, every consumer pulls a versioned snapshot and validates against it. Drift becomes a CI failure, not a production incident.