Expand SnowBridge to own ingest + in-process transform; drop dbt

SnowBridge now owns machine-data ingest, in-process .NET transformation,
and direct writes to curated tables in Snowflake. Collapses the previous
ingest/transform split into a single service; no dbt, no external
orchestrator, no Snowflake landing tier. Keeps the in-house .NET pattern
consistent with ScadaBridge and OtOpcUa.

The "Snowflake dbt Transform Layer" roadmap workstream merges into
SnowBridge (7 → 6 workstreams); Year 2 canonical-state-based OEE moves
with it. Canonical model still has three surfaces — the third is
renamed from "dbt curated layer" to "SnowBridge curated layer in
Snowflake"; mechanics unchanged.
This commit is contained in:
Joseph Doherty
2026-04-24 14:53:16 -04:00
parent 22a86974f6
commit 98bf2d0da4
15 changed files with 130 additions and 124 deletions

View File

@@ -133,7 +133,7 @@ These get resolved during implementation, not during design:
1. **Mermaid rendering in the Claude Code environment.** Unknown until I try. Fallback is manual rendering at mermaid.live; neither path breaks the design.
2. **Whether `document-skills:pptx` can produce a 3-column layout** (needed for slide 5: Three Pillars) and a 2-column layout (needed for slide 16: Open Coordination Items). If not, the spec falls back to single-column with visual separation.
3. **Table overflow behavior on slide 13.** The 7×3 roadmap grid with truncated cell content should fit one slide, but if it overflows, the spec needs a fallback: either shrink text or split across two slides.
3. **Table overflow behavior on slide 13.** The 6×3 roadmap grid with truncated cell content should fit one slide, but if it overflows, the spec needs a fallback: either shrink text or split across two slides.
4. **First-pass theme quality.** I'll use the theme-factory default; the first output becomes the visual baseline. If it looks wrong, section 3's "visual style" line is where the override goes.
## Task list

View File

@@ -123,7 +123,7 @@ Expected: `DESIGN.md IMPLEMENTATION-PLAN.md diagrams generated` (README, spec
**Content requirements:**
- Mermaid `flowchart LR` (left-to-right)
- Nodes match `goal-state.md` line 77 exactly: Equipment → OtOpcUa → System Platform/Ignition → ScadaBridge → Redpanda → SnowBridge → Snowflake → dbt → Power BI
- Nodes: Equipment → OtOpcUa → System Platform/Ignition → ScadaBridge → Redpanda → SnowBridge (ingest + in-process transform) → Snowflake (curated layer) → Power BI
- IT↔OT boundary marker between ScadaBridge and Redpanda (Redpanda is IT-adjacent from ScadaBridge's central crossing)
**Verification:**

View File

@@ -86,7 +86,7 @@ Diagrams are **hand-drawn source files** committed to [`diagrams/`](diagrams/),
The spec currently references two diagrams:
- `diagrams/architecture-layers.png` — the 4-layer goal-state architecture stack (Equipment → OtOpcUa → SCADA → ScadaBridge → Enterprise IT), with the IT↔OT boundary marked.
- `diagrams/end-to-end-flow.png` — the left-to-right data flow for one tag, matching `goal-state.md` line 77 exactly (Equipment → OtOpcUa → System Platform/Ignition → ScadaBridge → Redpanda → SnowBridge → Snowflake → dbt → Power BI).
- `diagrams/end-to-end-flow.png` — the left-to-right data flow for one tag (Equipment → OtOpcUa → System Platform/Ignition → ScadaBridge → Redpanda → SnowBridge → Snowflake → Power BI). SnowBridge owns ingest + in-process transform; no separate dbt hop.
Both are **not yet authored.** On first regeneration, Claude will either author the `.mmd` sources and attempt to render them, or flag this as a manual step in the run log. Until the PNGs exist, the corresponding slides (slides 8 and 9 in the deck) will have placeholder boxes.

View File

@@ -107,7 +107,7 @@ The Layered Architecture text diagram in `goal-state.md` (and any similar text d
Markdown tables render as PDF tables with row-level borders and header-row emphasis. Tables that exceed one page split cleanly at row boundaries, with the header row repeated at the top of each continuation page. `document-skills:pdf` handles this natively.
Specific large tables to expect:
- [`../roadmap.md`](../roadmap.md) → **The grid** (7 workstreams × 3 years) — likely spans 23 pages
- [`../roadmap.md`](../roadmap.md) → **The grid** (6 workstreams × 3 years) — likely spans 23 pages
- [`../current-state/legacy-integrations.md`](../current-state/legacy-integrations.md) → per-row integration detail tables (one per integration)
- [`../current-state/equipment-protocol-survey.md`](../current-state/equipment-protocol-survey.md) → field schema table, classification table, rollup views

View File

@@ -124,13 +124,13 @@ If `document-skills:pptx` cannot render a requested layout:
| **Source** | [`../goal-state.md`](../goal-state.md) → **OtOpcUa — the unified site-level OPC UA layer** |
| **Population** | 6 bullets: (1) single sanctioned OPC UA access point per site, one session per equipment; (2) two namespaces — equipment + System Platform (absorbs LmxOpcUa); (3) clustered, co-located on existing System Platform nodes; (4) hybrid driver strategy — proactive core library + on-demand long-tail; (5) OPC UA-native auth (UserName + standard security modes, inherited from LmxOpcUa); (6) tiered cutover — ScadaBridge → Ignition → System Platform IO across Years 13. |
## Slide 11 — Analytics Stack: SnowBridge, Snowflake, dbt
## Slide 11 — Analytics Stack: SnowBridge + Snowflake
| Property | Value |
|---|---|
| **Layout** | Content (bulleted) |
| **Source** | [`../goal-state.md`](../goal-state.md) → **SnowBridge** + **Aveva Historian → Snowflake** + **Snowflake-side transform tooling** |
| **Population** | 6 bullets: (1) SnowBridge — custom-built machine-data-to-Snowflake upload service; (2) source abstraction — Aveva Historian SQL in Year 1, Redpanda/ScadaBridge in Year 2; (3) governed selection with blast-radius approval workflow; (4) dbt curated layers, orchestrator out of scope; (5) ≤15-minute analytics SLO; (6) one "not possible before" AI/analytics use case in production by end of plan (pillar 2 gate). |
| **Population** | 6 bullets: (1) SnowBridge — custom-built .NET service owning **ingest + in-process transform + curated-table write** into Snowflake; (2) source abstraction — Aveva Historian SQL in Year 1, Redpanda/ScadaBridge in Year 2; (3) governed selection with blast-radius approval workflow; (4) **no dbt, no external orchestrator, no Snowflake landing tier** — transforms live in the service; (5) ≤15-minute analytics SLO; (6) one "not possible before" AI/analytics use case in production by end of plan (pillar 2 gate). |
## Slide 12 — Redpanda EventHub: the async backbone
@@ -146,16 +146,16 @@ If `document-skills:pptx` cannot render a requested layout:
|---|---|
| **Layout** | Table — 7 rows × 3 columns (+ workstream name column = 4 columns total) |
| **Source** | [`../roadmap.md`](../roadmap.md) → **The grid** |
| **Population** | Render the 7×3 roadmap grid as a PPTX table. **Truncate** each cell to the **single most important commitment** (not the full cell text, which would overflow). Workstream column: full name. Year columns: ~10-word headline per cell. Color-code cells by pillar if the theme supports it. |
| **Fallback** | If the 7×3 table doesn't fit one slide at readable type size, split across two slides: workstreams 14 on slide 13a (OtOpcUa, Redpanda, SnowBridge, dbt), workstreams 57 on slide 13b (ScadaBridge Extensions, Site Onboarding, Legacy Retirement). Label slides 13 and 14, renumber subsequent slides. |
| **Population** | Render the 6×3 roadmap grid as a PPTX table. **Truncate** each cell to the **single most important commitment** (not the full cell text, which would overflow). Workstream column: full name. Year columns: ~10-word headline per cell. Color-code cells by pillar if the theme supports it. |
| **Fallback** | If the 6×3 table doesn't fit one slide at readable type size, split across two slides: workstreams 13 on slide 13a (OtOpcUa, Redpanda, SnowBridge), workstreams 46 on slide 13b (ScadaBridge Extensions, Site Onboarding, Legacy Retirement). Label slides 13 and 14, renumber subsequent slides. |
## Slide 14 — Year 1 Focus
| Property | Value |
|---|---|
| **Layout** | Content (bulleted) |
| **Source** | [`../roadmap.md`](../roadmap.md) → the Year 1 column across all 7 workstreams |
| **Population** | 7 bullets, one per workstream, ordered by prerequisite position: (1) OtOpcUa — evolve LmxOpcUa, protocol survey, deploy to every site, begin tier-1 cutover; (2) Redpanda — stand up central cluster, schema registry, initial topics; (3) SnowBridge — design + first source adapter (Historian SQL) with filtered flow; (4) dbt — scaffold project, first curated model; (5) ScadaBridge Extensions — deadband publishing + EventHub producer; (6) Site Onboarding — document lightweight onboarding pattern (no new sites Year 1); (7) Legacy Retirement — populate inventory (done), retire first integration as pattern-proving exercise. |
| **Source** | [`../roadmap.md`](../roadmap.md) → the Year 1 column across all 6 workstreams |
| **Population** | 6 bullets, one per workstream, ordered by prerequisite position: (1) OtOpcUa — evolve LmxOpcUa, deploy to every site, begin tier-1 cutover, UNS hierarchy snapshot walk; (2) Redpanda — stand up central cluster, schema registry, initial topics, publish canonical model v1; (3) SnowBridge — design + first source adapter (Historian SQL) with filtered ingest + in-process transform + first curated tables aligned to canonical model; (4) ScadaBridge Extensions — deadband publishing + EventHub producer; (5) Site Onboarding — document lightweight onboarding pattern (no new sites Year 1); (6) Legacy Retirement — populate inventory (done), retire first integration as pattern-proving exercise. |
| **Rules** | **Exceeds the 6-bullet truncation rule.** 7 bullets here is intentional because each bullet represents one workstream's Year 1 commitment — dropping one would misrepresent the plan. Keep all 7, tighten wording to ≤10 words per bullet. |
## Slide 15 — Pillar 3: Legacy Retirement (3 → 0)
@@ -172,7 +172,7 @@ If `document-skills:pptx` cannot render a requested layout:
|---|---|
| **Layout** | 2-column content (fallback: single column with horizontal rule) |
| **Source** | [`../goal-state.md`](../goal-state.md) → **Strategic Considerations (Adjacent Asks)** |
| **Population** | **Left column — Digital twin (scope: two access-control patterns):** 4 bullets: (1) Scope is definitive — not a committed workstream, not a new component; (2) Pattern 1 — environment-lifecycle promotion without reconfiguration (ACL flip on write authority); (3) Pattern 2 — safe read-only consumption for KPI / monitoring systems (structural zero-write-path guarantee); (4) Both patterns are delivered by already-committed architecture (OtOpcUa ACL model + canonical model + single-connection-per-equipment). **Right column — BOBJ → Power BI:** 4 bullets: (1) In-flight reporting initiative, not owned by this plan; (2) Three consumption paths analyzed (Snowflake dbt / Historian direct / both); (3) Recommended position: Path C — hybrid, with Path A as strategic direction; (4) Next: schedule coordination conversation with reporting team — 8 questions ready in `goal-state.md`. |
| **Population** | **Left column — Digital twin (scope: two access-control patterns):** 4 bullets: (1) Scope is definitive — not a committed workstream, not a new component; (2) Pattern 1 — environment-lifecycle promotion without reconfiguration (ACL flip on write authority); (3) Pattern 2 — safe read-only consumption for KPI / monitoring systems (structural zero-write-path guarantee); (4) Both patterns are delivered by already-committed architecture (OtOpcUa ACL model + canonical model + single-connection-per-equipment). **Right column — BOBJ → Power BI:** 4 bullets: (1) In-flight reporting initiative, not owned by this plan; (2) Three consumption paths analyzed (SnowBridge curated layer in Snowflake / Historian direct / both); (3) Recommended position: Path C — hybrid, with Path A as strategic direction; (4) Next: schedule coordination conversation with reporting team — 8 questions ready in `goal-state.md`. |
## Slide 17 — Non-Goals
@@ -189,7 +189,7 @@ If `document-skills:pptx` cannot render a requested layout:
|---|---|
| **Layout** | Content (bulleted) |
| **Source** | [`../status.md`](../status.md) → **Top pending items** + inferred from [`../roadmap.md`](../roadmap.md) → Year 1 |
| **Population** | 4 bullets: (1) Sponsor confirmation + Year 1 funding commitment; (2) Named owners for each of the 7 workstreams (build team alignment); (3) Power BI coordination conversation with reporting team — schedule; (4) UNS hierarchy snapshot walk owner named (Q1Q2 Year 1 prerequisite for canonical model v1 publication). |
| **Population** | 4 bullets: (1) Sponsor confirmation + Year 1 funding commitment; (2) Named owners for each of the 6 workstreams (build team alignment); (3) Power BI coordination conversation with reporting team — schedule; (4) UNS hierarchy snapshot walk owner named (Q1Q2 Year 1 prerequisite for canonical model v1 publication). |
| **Notes** | This is the closer slide. Each bullet should be a discrete ask with a clear "who needs to do what" so the audience leaves with action. |
---