Files
infra/stacks/sql/README.md

37 lines
1.4 KiB
Markdown
Raw Normal View History

# sql
feat: SQL=postgres, nginx+certbot, MQTT split, ML stacks, gitea HTTPS-only, gemaal1 site Round-2 changes locking in scaffold-phase decisions and adding ML/notebook stacks. Locked decisions - sql: postgres 16-alpine (was TBD); init.d/ mount for per-app DB provisioning - nginx-proxy: stock nginx + certbot sidecar (was nginx:alpine TODO). Chose stock over nginxproxy/nginx-proxy because stream{} is required for MQTT-TLS reverse-proxy on tcp/8883 to rabbitmq:1883. - gitea: HTTPS-only (DISABLE_SSH=true). No SSH port published. MQTT split - Remove stacks/mqtt placeholder. - Add stacks/rabbitmq — general-purpose broker (AMQP + MQTT plugin), used at both cloud and edge. External MQTT clients reach cloud broker via nginx stream-proxy on 8883. - Add stacks/mosquitto — reserved for the FROST (SensorThings) stack only. Cloud-only. Internal to its own stack; no external ingress. ML / notebooks (cloud-only) - stacks/mlflow — experiment tracking + model registry. Postgres backend on sql stack; local volume for artifacts (S3/MinIO is a TODO). - stacks/jupyterhub — multi-user notebook server. DockerSpawner via mounted docker.sock; users spawn into cloud-app network so they can reach mlflow, influxdb (via grafana), rabbitmq. Sites - sites/gemaal1 — first edge deployment scaffold. Site-local override template for binding nginx to PLANT_LAN_IP. Docs - README + docs/architecture.md updated: stacks table now lists 15 stacks, ingress + attachment tables reflect mlflow/jupyterhub, TLS strategy section locked, MQTT-split section added, Gitea HTTPS-only noted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:22:46 +02:00
Central configuration database — the "single point of config" backing Keycloak, Gitea, MLflow, and any stack that needs a relational store. **Cloud-only.**
feat: SQL=postgres, nginx+certbot, MQTT split, ML stacks, gitea HTTPS-only, gemaal1 site Round-2 changes locking in scaffold-phase decisions and adding ML/notebook stacks. Locked decisions - sql: postgres 16-alpine (was TBD); init.d/ mount for per-app DB provisioning - nginx-proxy: stock nginx + certbot sidecar (was nginx:alpine TODO). Chose stock over nginxproxy/nginx-proxy because stream{} is required for MQTT-TLS reverse-proxy on tcp/8883 to rabbitmq:1883. - gitea: HTTPS-only (DISABLE_SSH=true). No SSH port published. MQTT split - Remove stacks/mqtt placeholder. - Add stacks/rabbitmq — general-purpose broker (AMQP + MQTT plugin), used at both cloud and edge. External MQTT clients reach cloud broker via nginx stream-proxy on 8883. - Add stacks/mosquitto — reserved for the FROST (SensorThings) stack only. Cloud-only. Internal to its own stack; no external ingress. ML / notebooks (cloud-only) - stacks/mlflow — experiment tracking + model registry. Postgres backend on sql stack; local volume for artifacts (S3/MinIO is a TODO). - stacks/jupyterhub — multi-user notebook server. DockerSpawner via mounted docker.sock; users spawn into cloud-app network so they can reach mlflow, influxdb (via grafana), rabbitmq. Sites - sites/gemaal1 — first edge deployment scaffold. Site-local override template for binding nginx to PLANT_LAN_IP. Docs - README + docs/architecture.md updated: stacks table now lists 15 stacks, ingress + attachment tables reflect mlflow/jupyterhub, TLS strategy section locked, MQTT-split section added, Gitea HTTPS-only noted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:22:46 +02:00
- **Engine**: postgres 16-alpine
- **Network**: `data` only (no internet egress)
feat: SQL=postgres, nginx+certbot, MQTT split, ML stacks, gitea HTTPS-only, gemaal1 site Round-2 changes locking in scaffold-phase decisions and adding ML/notebook stacks. Locked decisions - sql: postgres 16-alpine (was TBD); init.d/ mount for per-app DB provisioning - nginx-proxy: stock nginx + certbot sidecar (was nginx:alpine TODO). Chose stock over nginxproxy/nginx-proxy because stream{} is required for MQTT-TLS reverse-proxy on tcp/8883 to rabbitmq:1883. - gitea: HTTPS-only (DISABLE_SSH=true). No SSH port published. MQTT split - Remove stacks/mqtt placeholder. - Add stacks/rabbitmq — general-purpose broker (AMQP + MQTT plugin), used at both cloud and edge. External MQTT clients reach cloud broker via nginx stream-proxy on 8883. - Add stacks/mosquitto — reserved for the FROST (SensorThings) stack only. Cloud-only. Internal to its own stack; no external ingress. ML / notebooks (cloud-only) - stacks/mlflow — experiment tracking + model registry. Postgres backend on sql stack; local volume for artifacts (S3/MinIO is a TODO). - stacks/jupyterhub — multi-user notebook server. DockerSpawner via mounted docker.sock; users spawn into cloud-app network so they can reach mlflow, influxdb (via grafana), rabbitmq. Sites - sites/gemaal1 — first edge deployment scaffold. Site-local override template for binding nginx to PLANT_LAN_IP. Docs - README + docs/architecture.md updated: stacks table now lists 15 stacks, ingress + attachment tables reflect mlflow/jupyterhub, TLS strategy section locked, MQTT-split section added, Gitea HTTPS-only noted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:22:46 +02:00
- **Volume**: `sql-data` (PGDATA)
feat(cloud): harden nginx-proxy + sql foundation; HTTP-01 interim cert plan Wire up the three foundation stacks (nginx-proxy, sql, portainer) in cloud/compose.yml and add real configs for the first two. nginx-proxy - Base nginx.conf with http + stream contexts, modern TLS profile, client_max_body_size baseline for gitea LFS / mlflow artifacts. - Vhosts under conf.d/: grafana, gitea, keycloak, nodered, mlflow, jupyter, portainer (HTTPS upstream), rabbitmq, jenkins. WebSocket upgrade headers where needed (grafana live, node-red editor, jupyterhub kernels, jenkins agents). - conf.d/00-default.conf serves /.well-known/acme-challenge/ on :80 and 301-redirects everything else. - stream.d/mqtt.conf terminates MQTT-TLS at 8883, proxies to rabbitmq:1883 internally. - All vhosts reference /etc/letsencrypt/live/infra/* — a stable path via certbot --cert-name infra, so the wildcard migration changes nothing in the vhost files. - README documents: HTTP-01 SAN interim during Versio period → DNS-01 wildcard via certbot-dns-transip after migration; bootstrap procedure (self-signed fallback → real cert issuance → reload). sql - config/init.d/01-databases.sh provisions gitea/keycloak/mlflow databases + roles on first start. Idempotent only via fresh data volume — change the script after first run requires manual psql or a volume wipe. - compose env extended with GITEA_DB_PASSWORD, KEYCLOAK_DB_PASSWORD, MLFLOW_DB_PASSWORD. cloud - include: now wires nginx-proxy + sql + portainer. Other stacks stay commented for future rounds. - .env.example adds KEYCLOAK_DB_PASSWORD and sensible defaults (LETSENCRYPT_EMAIL, GRAFANA_ROOT_URL, KEYCLOAK_HOSTNAME, GITEA_ROOT_URL, POSTFIX_FROM_DOMAIN all pointing at wbd-rd.nl). - Operator note inline: bring portainer's standalone instance down before deploying via cloud compose; comment its ports: block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:43:35 +02:00
- **Init scripts**: `config/init.d/*.sh` runs once on first container start
## Per-app databases
On first start, `config/init.d/01-databases.sh` provisions:
| Database | Owner role | Used by |
|---|---|---|
| `gitea` | `gitea` | gitea stack |
| `keycloak` | `keycloak` | keycloak stack |
| `mlflow` | `mlflow` | mlflow stack |
Passwords come from env vars (`GITEA_DB_PASSWORD`, `KEYCLOAK_DB_PASSWORD`, `MLFLOW_DB_PASSWORD`) which must be set in the cloud `.env` *before* first start.
**Important**: init scripts only run when `sql-data` is empty. Changing the script after first start has no effect until the volume is wiped. To add a new app DB later, connect with `psql` and create it manually, then update this script for fresh deploys.
## Reset / re-init
```bash
docker compose down
docker volume rm cloud_sql-data # ⚠ destroys all data
docker compose up -d
```
## TODO
- Backup strategy (pg_dump cron sidecar vs streaming replica vs WAL archiving to MinIO)
- Per-app least-privilege grants (currently each role owns its DB only — fine for now)
- Monitoring (postgres_exporter for Prometheus when observability stack lands)