ScadaBridge/docs/test_infra/test_infra_db.md

# Test Infrastructure: MS SQL 2022 Database

## Overview

The test database uses Microsoft SQL Server 2022 Developer Edition running in Docker. It provides two empty databases for ScadaBridge — schema creation is handled by EF Core migrations at application startup (dev mode).

## Image & Ports

- **Image**: `mcr.microsoft.com/mssql/server:2022-latest`
- **Port**: 1433
- **Edition**: Developer (free, full-featured)

## Credentials

| Account | Username | Password | Purpose |
|---------|----------|----------|---------|
| SA | `sa` | `ScadaBridge_Dev1#` | Server admin (setup only) |
| App | `scadabridge_app` | `ScadaBridge_Dev1#` | Application login (db_owner on both databases) |

## Databases

| Database | Purpose |
|----------|---------|
| `ScadaBridgeConfig` | Configuration Database component — templates, deployments, users, audit log |
| `ScadaBridgeMachineData` | Machine/operational data storage |

Both databases are created by `infra/mssql/setup.sql`. EF Core migrations populate the `ScadaBridgeConfig` schema. The `ScadaBridgeMachineData` database is seeded with sample tables and stored procedures by `infra/mssql/machinedata_seed.sql`.

### Ephemeral per-test-class databases

`MsSqlMigrationFixture` (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/`) creates a
throwaway `ScadaBridgeAuditMigTest_<guid>` database per test class, applies the full EF migration chain
to it, and drops it on teardown. Both `ConfigurationDatabase.Tests` and `AuditLog.Tests` use it — the
latter by project reference, so there is one fixture implementation, not a copy. Seeing a handful of
these databases come and go while tests run is normal; seeing them survive a run is not (see below).

Three behaviours are worth knowing when you look at this server during a test run:

- **Concurrent create+migrate is throttled machine-wide.** Fixtures take a slot from a counting
  semaphore built on `sp_getapplock` in `master` (resources `ScadaBridgeTestMigrate/0..N-1`) and hold it
  across `CREATE DATABASE` + the migration chain. The default is 4 slots; override with
  `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`. An xunit collection attribute cannot do this job — a
  solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
  to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
  the container and individual DDL statements blew past EF's default 30-second command timeout,
  failing whole test classes at fixture construction (deferred-work register row 82). The
  *in-process* half of the same problem is bounded separately, by a `maxParallelThreads: 4` cap in
  each MSSQL-backed suite's `xunit.runner.json` — deliberately the same number, so the two limits
  agree. See [test_infra.md → Test Parallelism Policy](test_infra.md#test-parallelism-policy).
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
  the server is busy but progressing, so the correct response to a slow statement is to wait.
- **Strays are swept.** Each test process drops, once at start and best-effort, any
  `ScadaBridgeAuditMigTest_*` database older than 2 hours, so a run killed before teardown does not
  leave databases accumulating. To clear them by hand:

```bash
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
  -Q "SELECT name, create_date FROM sys.databases WHERE name LIKE 'ScadaBridgeAuditMigTest_%'"
```

Overriding the server the fixture targets (CI, a non-default port) is done with
`SCADABRIDGE_MSSQL_TEST_CONN`. When no server answers, the fixture reports `Available = false` and the
tests report as **Skipped**, not passed.

## Data Persistence

SQL data is stored in the named Docker volume `scadabridge-mssql-data`. Data survives container restarts and `docker compose down`. To reset the database completely:

```bash
docker compose down -v
docker compose up -d
# Re-run setup.sql after the container starts
```

## First-Time Setup

After the first `docker compose up -d`, run the setup script:

```bash
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
  -i /docker-entrypoint-initdb.d/setup.sql
```

This creates the databases and the `scadabridge_app` login. Then seed the Machine Data database with sample tables, stored procedures, and data:

```bash
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
  -i /docker-entrypoint-initdb.d/machinedata_seed.sql
```

You only need to run these once (or again after deleting the volume).

## Connection Strings

For `appsettings.Development.json`:

```
Server=localhost,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true
```

```
Server=localhost,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true
```

## Verification

1. Check the container is running:

```bash
docker ps --filter name=scadabridge-mssql
```

2. Query using `sqlcmd` inside the container:

```bash
docker exec -it scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
  -Q "SELECT name FROM sys.databases"
```

3. Verify the app login:

```bash
docker exec -it scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U scadabridge_app -P 'ScadaBridge_Dev1#' -C \
  -d ScadaBridgeConfig \
  -Q "SELECT DB_NAME()"
```

## CLI Tool

The `infra/tools/mssql_tool.py` script provides a convenient CLI for interacting with the SQL Server.

**Install dependencies** (one-time):
```bash
pip install -r infra/tools/requirements.txt
```

**Commands**:

```bash
# Check connectivity and verify expected databases exist
python infra/tools/mssql_tool.py check

# Run the first-time setup script (uses autocommit mode for CREATE DATABASE)
python infra/tools/mssql_tool.py setup --script infra/mssql/setup.sql

# List tables in a database
python infra/tools/mssql_tool.py tables --database ScadaBridgeConfig

# Run an ad-hoc query
python infra/tools/mssql_tool.py query --database ScadaBridgeConfig --sql "SELECT name FROM sys.tables"
```

Use `--host`, `--port`, `--user`, `--password` to override defaults (localhost:1433, sa, ScadaBridge_Dev1#). Run with `--help` for full usage.

## Machine Data Tables & Stored Procedures

The `machinedata_seed.sql` script creates the following in `ScadaBridgeMachineData`:

**Tables**:

| Table | Description | Sample Data |
|-------|-------------|-------------|
| `TagHistory` | Time-series tag values from OPC UA / custom protocols | Pressure, flow, level, temperature, speed readings for SiteA/SiteB |
| `ProductionCounts` | Shift/line production totals (good, reject, efficiency) | 3 days of 2-shift data across 2 sites |
| `EquipmentEvents` | State changes, faults, maintenance, alarm events | Pump faults, belt inspections, batch starts |
| `BatchRecords` | Production batch tracking (start, complete, abort) | 5 batches including one in-progress |
| `AlarmHistory` | Historical alarm activations, acks, and clears | Active, acknowledged, and cleared alarms |

**Stored Procedures**:

| Procedure | Description |
|-----------|-------------|
| `usp_GetTagHistory` | Get tag values for a tag path within a date range |
| `usp_GetProductionSummary` | Aggregate production by line over a date range |
| `usp_InsertBatchRecord` | Insert a new batch (for `CachedWrite` testing) |
| `usp_CompleteBatch` | Complete or abort a batch |
| `usp_GetEquipmentEvents` | Get recent equipment events with optional filters |
| `usp_GetActiveAlarms` | Get active/acknowledged alarms by severity |

## Relevance to ScadaBridge Components

- **Configuration Database** — primary consumer; EF Core context targets `ScadaBridgeConfig`.
- **Deployment Manager** — reads/writes deployment records in `ScadaBridgeConfig`.
- **Template Engine** — reads/writes template definitions in `ScadaBridgeConfig`.
- **Security & Auth** — user/role data stored in `ScadaBridgeConfig`.
- **External System Gateway** — scripts use `Database.Connection("machineDataConnection")` to query `ScadaBridgeMachineData` tables and stored procedures.
- **Site Runtime** — scripts call stored procedures via `Database.Connection()` and `Database.CachedWrite()` for batch recording and data queries.
- **Inbound API** — methods can query machine data via named database connections.

## Notes

- The `sa` password must meet SQL Server complexity requirements (uppercase, lowercase, digit, special character, 8+ characters).
- If the container fails to start, check Docker has at least 2GB RAM allocated (SQL Server minimum requirement).
- The setup script is idempotent — safe to run multiple times.